2024-04-19T01:49:07.599 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-19T01:49:07.599 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} --name teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi --verbose --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662749 -- /home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662749/orig.config.yaml 2024-04-19T01:49:07.641 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-19T01:49:07.739 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662749 branch: squid description: rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} email: ceph-qa@ceph.com first_in_suite: false job_id: '7662749' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 osd_object_clean_region_max_num_intervals: 1000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd heartbeat use min delay socket: true osd max backfills: 6 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - slow request - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED sha1: 1a6a3e742fd81c6d70be70552499bff767652c32 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 1a6a3e742fd81c6d70be70552499bff767652c32 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: d0bf63dd179c05751e4be49477389aa290c73089 owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - mon.a - - osd.4 - osd.5 - osd.6 - osd.7 - mon.b - mgr.x - - osd.8 - osd.9 - osd.10 - osd.11 - mon.c seed: 6512 sha1: 1a6a3e742fd81c6d70be70552499bff767652c32 sleep_before_teardown: 0 subset: 28391/100000 suite: rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_d0bf63dd179c05751e4be49477389aa290c73089/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: d0bf63dd179c05751e4be49477389aa290c73089 targets: smithi059.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC78c5xWh8jIJwnkSb8HC90fPq7HoC3QQoqJO/0XfrGGer1Bajr3tHVTzQJNSKm5ca0Gz9ylZvP7ATr1ol+l1qc= smithi081.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBEUm2ng7ce4gftKKX9BeQN3+wp1ILw30pdA4UEdIBM+YzbjV+UjG81RQlgZTiXAbNkYg+WqcgKMkc8IbJ8P9F8= smithi179.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPTcNsLadPM7gOyBxe0Cie1W6T30xpnh0+lOPAZfC9BHHxJkMDg0ppgUuuau/+lqCPD1YVV5ugOmKDCdt7VKHns= tasks: - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: aggressive_pg_num_changes: false chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 min_in: 8 timeout: 1200 - rados: clients: - client.0 ec_pool: true erasure_code_profile: crush-failure-domain: osd k: 4 l: 3 m: 2 name: lrcprofile plugin: lrc objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 400 write_append_excl: false teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-18_21:00:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-19T01:49:07.739 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_d0bf63dd179c05751e4be49477389aa290c73089/qa; will attempt to use it 2024-04-19T01:49:07.740 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_d0bf63dd179c05751e4be49477389aa290c73089/qa/tasks 2024-04-19T01:49:07.740 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-19T01:49:07.742 INFO:teuthology.task.internal:Checking packages... 2024-04-19T01:49:07.768 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '1a6a3e742fd81c6d70be70552499bff767652c32' 2024-04-19T01:49:07.768 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-19T01:49:07.768 INFO:teuthology.packaging:ref: None 2024-04-19T01:49:07.769 INFO:teuthology.packaging:tag: None 2024-04-19T01:49:07.769 INFO:teuthology.packaging:branch: squid 2024-04-19T01:49:07.769 INFO:teuthology.packaging:sha1: 1a6a3e742fd81c6d70be70552499bff767652c32 2024-04-19T01:49:07.769 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-04-19T01:49:07.989 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2401-gd0bf63dd-1jammy 2024-04-19T01:49:07.990 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-19T01:49:07.997 INFO:teuthology.task.internal:no buildpackages task found 2024-04-19T01:49:07.998 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-19T01:49:08.006 INFO:teuthology.task.internal:Saving configuration 2024-04-19T01:49:08.025 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-19T01:49:08.032 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-19T01:49:08.055 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662749', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-19 01:43:28.990481', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMtC9bPrsSD2crzrydOihA38EViaWH4vA3xv7ROlv3a2'} 2024-04-19T01:49:08.077 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi081.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662749', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-19 01:43:28.991824', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINYr08Lpe3R9GJYA8PoP6hxdOsakT5MlbjyS8QC5wm0g'} 2024-04-19T01:49:08.099 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi179.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662749', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-19 01:43:28.992922', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIL5ixkZDMR82gZyy538Nyem3nQdax+rPgJyUTGLXr3lW'} 2024-04-19T01:49:08.099 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-19T01:49:08.106 INFO:teuthology.task.internal:roles: ubuntu@smithi059.front.sepia.ceph.com - ['osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'mon.a'] 2024-04-19T01:49:08.106 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['osd.4', 'osd.5', 'osd.6', 'osd.7', 'mon.b', 'mgr.x'] 2024-04-19T01:49:08.106 INFO:teuthology.task.internal:roles: ubuntu@smithi179.front.sepia.ceph.com - ['osd.8', 'osd.9', 'osd.10', 'osd.11', 'mon.c'] 2024-04-19T01:49:08.107 INFO:teuthology.run_tasks:Running task console_log... 2024-04-19T01:49:08.201 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9e1b40d280>, signals=[15]) 2024-04-19T01:49:08.202 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-19T01:49:08.208 INFO:teuthology.task.internal:Opening connections... 2024-04-19T01:49:08.209 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi059.front.sepia.ceph.com 2024-04-19T01:49:08.210 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:49:08.294 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi081.front.sepia.ceph.com 2024-04-19T01:49:08.295 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:49:08.375 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi179.front.sepia.ceph.com 2024-04-19T01:49:08.376 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:49:08.454 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-19T01:49:08.462 DEBUG:teuthology.orchestra.run.smithi059:> uname -m 2024-04-19T01:49:08.468 INFO:teuthology.orchestra.run.smithi059.stdout:x86_64 2024-04-19T01:49:08.468 DEBUG:teuthology.orchestra.run.smithi059:> cat /etc/os-release 2024-04-19T01:49:08.516 INFO:teuthology.orchestra.run.smithi059.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-19T01:49:08.516 INFO:teuthology.orchestra.run.smithi059.stdout:NAME="Ubuntu" 2024-04-19T01:49:08.516 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION_ID="22.04" 2024-04-19T01:49:08.516 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-19T01:49:08.516 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION_CODENAME=jammy 2024-04-19T01:49:08.516 INFO:teuthology.orchestra.run.smithi059.stdout:ID=ubuntu 2024-04-19T01:49:08.516 INFO:teuthology.orchestra.run.smithi059.stdout:ID_LIKE=debian 2024-04-19T01:49:08.516 INFO:teuthology.orchestra.run.smithi059.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-19T01:49:08.517 INFO:teuthology.orchestra.run.smithi059.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-19T01:49:08.517 INFO:teuthology.orchestra.run.smithi059.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-19T01:49:08.517 INFO:teuthology.orchestra.run.smithi059.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-19T01:49:08.517 INFO:teuthology.orchestra.run.smithi059.stdout:UBUNTU_CODENAME=jammy 2024-04-19T01:49:08.517 INFO:teuthology.lock.ops:Updating smithi059.front.sepia.ceph.com on lock server 2024-04-19T01:49:08.551 DEBUG:teuthology.orchestra.run.smithi081:> uname -m 2024-04-19T01:49:08.557 INFO:teuthology.orchestra.run.smithi081.stdout:x86_64 2024-04-19T01:49:08.557 DEBUG:teuthology.orchestra.run.smithi081:> cat /etc/os-release 2024-04-19T01:49:08.605 INFO:teuthology.orchestra.run.smithi081.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-19T01:49:08.605 INFO:teuthology.orchestra.run.smithi081.stdout:NAME="Ubuntu" 2024-04-19T01:49:08.605 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION_ID="22.04" 2024-04-19T01:49:08.605 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-19T01:49:08.605 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION_CODENAME=jammy 2024-04-19T01:49:08.605 INFO:teuthology.orchestra.run.smithi081.stdout:ID=ubuntu 2024-04-19T01:49:08.605 INFO:teuthology.orchestra.run.smithi081.stdout:ID_LIKE=debian 2024-04-19T01:49:08.605 INFO:teuthology.orchestra.run.smithi081.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-19T01:49:08.605 INFO:teuthology.orchestra.run.smithi081.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-19T01:49:08.605 INFO:teuthology.orchestra.run.smithi081.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-19T01:49:08.605 INFO:teuthology.orchestra.run.smithi081.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-19T01:49:08.605 INFO:teuthology.orchestra.run.smithi081.stdout:UBUNTU_CODENAME=jammy 2024-04-19T01:49:08.606 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server 2024-04-19T01:49:08.625 DEBUG:teuthology.orchestra.run.smithi179:> uname -m 2024-04-19T01:49:08.631 INFO:teuthology.orchestra.run.smithi179.stdout:x86_64 2024-04-19T01:49:08.632 DEBUG:teuthology.orchestra.run.smithi179:> cat /etc/os-release 2024-04-19T01:49:08.679 INFO:teuthology.orchestra.run.smithi179.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-19T01:49:08.679 INFO:teuthology.orchestra.run.smithi179.stdout:NAME="Ubuntu" 2024-04-19T01:49:08.679 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION_ID="22.04" 2024-04-19T01:49:08.680 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-19T01:49:08.680 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION_CODENAME=jammy 2024-04-19T01:49:08.680 INFO:teuthology.orchestra.run.smithi179.stdout:ID=ubuntu 2024-04-19T01:49:08.680 INFO:teuthology.orchestra.run.smithi179.stdout:ID_LIKE=debian 2024-04-19T01:49:08.680 INFO:teuthology.orchestra.run.smithi179.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-19T01:49:08.680 INFO:teuthology.orchestra.run.smithi179.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-19T01:49:08.680 INFO:teuthology.orchestra.run.smithi179.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-19T01:49:08.680 INFO:teuthology.orchestra.run.smithi179.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-19T01:49:08.680 INFO:teuthology.orchestra.run.smithi179.stdout:UBUNTU_CODENAME=jammy 2024-04-19T01:49:08.680 INFO:teuthology.lock.ops:Updating smithi179.front.sepia.ceph.com on lock server 2024-04-19T01:49:08.700 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-19T01:49:08.709 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-19T01:49:08.714 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-19T01:49:08.714 DEBUG:teuthology.orchestra.run.smithi059:> test '!' -e /home/ubuntu/cephtest 2024-04-19T01:49:08.717 DEBUG:teuthology.orchestra.run.smithi081:> test '!' -e /home/ubuntu/cephtest 2024-04-19T01:49:08.720 DEBUG:teuthology.orchestra.run.smithi179:> test '!' -e /home/ubuntu/cephtest 2024-04-19T01:49:08.727 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-19T01:49:08.733 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-19T01:49:08.733 DEBUG:teuthology.orchestra.run.smithi059:> test -z $(ls -A /var/lib/ceph) 2024-04-19T01:49:08.765 DEBUG:teuthology.orchestra.run.smithi081:> test -z $(ls -A /var/lib/ceph) 2024-04-19T01:49:08.767 DEBUG:teuthology.orchestra.run.smithi179:> test -z $(ls -A /var/lib/ceph) 2024-04-19T01:49:08.782 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-19T01:49:08.850 INFO:teuthology.run_tasks:Running task kernel... 2024-04-19T01:49:08.864 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-19T01:49:08.865 INFO:teuthology.task.kernel:config {'osd.0': {'kdb': 1, 'sha1': 'distro'}, 'osd.4': {'kdb': 1, 'sha1': 'distro'}, 'osd.8': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-19T01:49:08.865 DEBUG:teuthology.orchestra.run.smithi059:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-19T01:49:08.865 DEBUG:teuthology.orchestra.run.smithi081:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-19T01:49:08.866 DEBUG:teuthology.orchestra.run.smithi179:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-19T01:49:08.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:49:08.872 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2024-04-19T01:49:08.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:49:08.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:49:08.873 DEBUG:teuthology.orchestra.run.smithi179:> uname -r 2024-04-19T01:49:08.874 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2024-04-19T01:49:08.920 INFO:teuthology.orchestra.run.smithi059.stdout:5.15.0-56-generic 2024-04-19T01:49:08.920 INFO:teuthology.task.kernel:Running kernel on smithi059: 5.15.0-56-generic 2024-04-19T01:49:08.920 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt-get clean 2024-04-19T01:49:08.922 INFO:teuthology.orchestra.run.smithi081.stdout:5.15.0-56-generic 2024-04-19T01:49:08.922 INFO:teuthology.orchestra.run.smithi179.stdout:5.15.0-56-generic 2024-04-19T01:49:08.922 INFO:teuthology.task.kernel:Running kernel on smithi081: 5.15.0-56-generic 2024-04-19T01:49:08.923 DEBUG:teuthology.orchestra.run.smithi081:> sudo apt-get clean 2024-04-19T01:49:08.923 INFO:teuthology.task.kernel:Running kernel on smithi179: 5.15.0-56-generic 2024-04-19T01:49:08.923 DEBUG:teuthology.orchestra.run.smithi179:> sudo apt-get clean 2024-04-19T01:49:09.046 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt-get update 2024-04-19T01:49:09.059 DEBUG:teuthology.orchestra.run.smithi179:> sudo apt-get update 2024-04-19T01:49:09.064 DEBUG:teuthology.orchestra.run.smithi081:> sudo apt-get update 2024-04-19T01:49:09.197 INFO:teuthology.orchestra.run.smithi059.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-19T01:49:09.209 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T01:49:09.209 INFO:teuthology.orchestra.run.smithi081.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-19T01:49:09.218 INFO:teuthology.orchestra.run.smithi081.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-19T01:49:09.245 INFO:teuthology.orchestra.run.smithi179.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T01:49:09.253 INFO:teuthology.orchestra.run.smithi179.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-19T01:49:09.253 INFO:teuthology.orchestra.run.smithi179.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-19T01:49:09.308 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T01:49:09.308 INFO:teuthology.orchestra.run.smithi081.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-19T01:49:09.342 INFO:teuthology.orchestra.run.smithi179.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-19T01:49:09.390 INFO:teuthology.orchestra.run.smithi059.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-19T01:49:09.435 INFO:teuthology.orchestra.run.smithi059.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,351 kB] 2024-04-19T01:49:09.447 INFO:teuthology.orchestra.run.smithi081.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,351 kB] 2024-04-19T01:49:09.476 INFO:teuthology.orchestra.run.smithi179.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,351 kB] 2024-04-19T01:49:09.520 INFO:teuthology.orchestra.run.smithi081.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,597 kB] 2024-04-19T01:49:09.547 INFO:teuthology.orchestra.run.smithi179.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [616 kB] 2024-04-19T01:49:09.550 INFO:teuthology.orchestra.run.smithi059.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [445 kB] 2024-04-19T01:49:09.557 INFO:teuthology.orchestra.run.smithi059.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [238 kB] 2024-04-19T01:49:09.560 INFO:teuthology.orchestra.run.smithi059.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-19T01:49:09.561 INFO:teuthology.orchestra.run.smithi059.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,680 kB] 2024-04-19T01:49:09.561 INFO:teuthology.orchestra.run.smithi081.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [445 kB] 2024-04-19T01:49:09.570 INFO:teuthology.orchestra.run.smithi081.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [238 kB] 2024-04-19T01:49:09.576 INFO:teuthology.orchestra.run.smithi081.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-19T01:49:09.576 INFO:teuthology.orchestra.run.smithi081.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,680 kB] 2024-04-19T01:49:09.595 INFO:teuthology.orchestra.run.smithi059.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-19T01:49:09.595 INFO:teuthology.orchestra.run.smithi059.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [281 kB] 2024-04-19T01:49:09.600 INFO:teuthology.orchestra.run.smithi059.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-19T01:49:09.600 INFO:teuthology.orchestra.run.smithi059.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [600 kB] 2024-04-19T01:49:09.613 INFO:teuthology.orchestra.run.smithi059.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-19T01:49:09.613 INFO:teuthology.orchestra.run.smithi081.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-19T01:49:09.613 INFO:teuthology.orchestra.run.smithi081.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [281 kB] 2024-04-19T01:49:09.617 INFO:teuthology.orchestra.run.smithi179.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [445 kB] 2024-04-19T01:49:09.621 INFO:teuthology.orchestra.run.smithi081.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-19T01:49:09.621 INFO:teuthology.orchestra.run.smithi081.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-19T01:49:09.626 INFO:teuthology.orchestra.run.smithi179.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [238 kB] 2024-04-19T01:49:09.633 INFO:teuthology.orchestra.run.smithi179.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-19T01:49:09.634 INFO:teuthology.orchestra.run.smithi179.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-19T01:49:09.635 INFO:teuthology.orchestra.run.smithi179.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,680 kB] 2024-04-19T01:49:09.642 INFO:teuthology.orchestra.run.smithi059.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-04-19T01:49:09.646 INFO:teuthology.orchestra.run.smithi059.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-19T01:49:09.650 INFO:teuthology.orchestra.run.smithi059.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-19T01:49:09.651 INFO:teuthology.orchestra.run.smithi059.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-19T01:49:09.654 INFO:teuthology.orchestra.run.smithi081.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [600 kB] 2024-04-19T01:49:09.659 INFO:teuthology.orchestra.run.smithi081.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-04-19T01:49:09.660 INFO:teuthology.orchestra.run.smithi081.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-19T01:49:09.661 INFO:teuthology.orchestra.run.smithi081.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-19T01:49:09.661 INFO:teuthology.orchestra.run.smithi081.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-19T01:49:09.681 INFO:teuthology.orchestra.run.smithi179.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,597 kB] 2024-04-19T01:49:09.681 INFO:teuthology.orchestra.run.smithi179.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [281 kB] 2024-04-19T01:49:09.682 INFO:teuthology.orchestra.run.smithi081.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [616 kB] 2024-04-19T01:49:09.684 INFO:teuthology.orchestra.run.smithi179.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-19T01:49:09.684 INFO:teuthology.orchestra.run.smithi179.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [600 kB] 2024-04-19T01:49:09.693 INFO:teuthology.orchestra.run.smithi179.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-19T01:49:09.703 INFO:teuthology.orchestra.run.smithi179.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-04-19T01:49:09.704 INFO:teuthology.orchestra.run.smithi179.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-19T01:49:09.705 INFO:teuthology.orchestra.run.smithi081.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [302 kB] 2024-04-19T01:49:09.705 INFO:teuthology.orchestra.run.smithi081.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-19T01:49:09.705 INFO:teuthology.orchestra.run.smithi081.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-19T01:49:09.710 INFO:teuthology.orchestra.run.smithi081.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-19T01:49:09.711 INFO:teuthology.orchestra.run.smithi081.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,806 kB] 2024-04-19T01:49:09.734 INFO:teuthology.orchestra.run.smithi179.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-19T01:49:09.734 INFO:teuthology.orchestra.run.smithi179.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-19T01:49:09.735 INFO:teuthology.orchestra.run.smithi179.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-19T01:49:09.735 INFO:teuthology.orchestra.run.smithi179.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-19T01:49:09.745 INFO:teuthology.orchestra.run.smithi081.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-19T01:49:09.745 INFO:teuthology.orchestra.run.smithi179.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [302 kB] 2024-04-19T01:49:09.745 INFO:teuthology.orchestra.run.smithi081.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [305 kB] 2024-04-19T01:49:09.754 INFO:teuthology.orchestra.run.smithi179.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-19T01:49:09.755 INFO:teuthology.orchestra.run.smithi179.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-19T01:49:09.756 INFO:teuthology.orchestra.run.smithi179.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,806 kB] 2024-04-19T01:49:09.787 INFO:teuthology.orchestra.run.smithi059.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-19T01:49:09.788 INFO:teuthology.orchestra.run.smithi059.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-19T01:49:09.788 INFO:teuthology.orchestra.run.smithi059.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-19T01:49:09.795 INFO:teuthology.orchestra.run.smithi081.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-19T01:49:09.796 INFO:teuthology.orchestra.run.smithi081.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-19T01:49:09.796 INFO:teuthology.orchestra.run.smithi179.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [305 kB] 2024-04-19T01:49:09.801 INFO:teuthology.orchestra.run.smithi179.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-19T01:49:09.801 INFO:teuthology.orchestra.run.smithi179.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [699 kB] 2024-04-19T01:49:09.810 INFO:teuthology.orchestra.run.smithi179.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-19T01:49:09.811 INFO:teuthology.orchestra.run.smithi081.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [699 kB] 2024-04-19T01:49:09.822 INFO:teuthology.orchestra.run.smithi179.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-19T01:49:09.823 INFO:teuthology.orchestra.run.smithi081.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-19T01:49:09.824 INFO:teuthology.orchestra.run.smithi179.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-19T01:49:09.825 INFO:teuthology.orchestra.run.smithi179.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-19T01:49:09.825 INFO:teuthology.orchestra.run.smithi179.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-19T01:49:09.826 INFO:teuthology.orchestra.run.smithi179.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-19T01:49:09.827 INFO:teuthology.orchestra.run.smithi081.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-19T01:49:09.828 INFO:teuthology.orchestra.run.smithi081.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-19T01:49:09.828 INFO:teuthology.orchestra.run.smithi081.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-19T01:49:09.829 INFO:teuthology.orchestra.run.smithi081.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-19T01:49:09.829 INFO:teuthology.orchestra.run.smithi081.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-19T01:49:09.829 INFO:teuthology.orchestra.run.smithi081.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-19T01:49:09.830 INFO:teuthology.orchestra.run.smithi081.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-19T01:49:09.835 INFO:teuthology.orchestra.run.smithi179.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-19T01:49:09.835 INFO:teuthology.orchestra.run.smithi179.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-19T01:49:09.836 INFO:teuthology.orchestra.run.smithi179.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-19T01:49:09.837 INFO:teuthology.orchestra.run.smithi179.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-19T01:49:09.842 INFO:teuthology.orchestra.run.smithi081.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-19T01:49:09.856 INFO:teuthology.orchestra.run.smithi179.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-19T01:49:09.856 INFO:teuthology.orchestra.run.smithi179.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-19T01:49:09.857 INFO:teuthology.orchestra.run.smithi179.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-19T01:49:09.858 INFO:teuthology.orchestra.run.smithi179.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-19T01:49:09.858 INFO:teuthology.orchestra.run.smithi179.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-19T01:49:09.865 INFO:teuthology.orchestra.run.smithi081.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-19T01:49:09.865 INFO:teuthology.orchestra.run.smithi081.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-19T01:49:09.865 INFO:teuthology.orchestra.run.smithi081.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-19T01:49:09.866 INFO:teuthology.orchestra.run.smithi081.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-19T01:49:09.867 INFO:teuthology.orchestra.run.smithi081.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-19T01:49:09.903 INFO:teuthology.orchestra.run.smithi059.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,597 kB] 2024-04-19T01:49:10.298 INFO:teuthology.orchestra.run.smithi059.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [616 kB] 2024-04-19T01:49:10.321 INFO:teuthology.orchestra.run.smithi059.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [302 kB] 2024-04-19T01:49:10.331 INFO:teuthology.orchestra.run.smithi059.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-19T01:49:10.332 INFO:teuthology.orchestra.run.smithi059.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,806 kB] 2024-04-19T01:49:10.416 INFO:teuthology.orchestra.run.smithi059.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-19T01:49:10.418 INFO:teuthology.orchestra.run.smithi059.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [305 kB] 2024-04-19T01:49:10.428 INFO:teuthology.orchestra.run.smithi059.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-19T01:49:10.593 INFO:teuthology.orchestra.run.smithi059.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [699 kB] 2024-04-19T01:49:11.177 INFO:teuthology.orchestra.run.smithi059.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-19T01:49:11.288 INFO:teuthology.orchestra.run.smithi059.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-19T01:49:11.308 INFO:teuthology.orchestra.run.smithi059.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-19T01:49:11.309 INFO:teuthology.orchestra.run.smithi059.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-19T01:49:11.313 INFO:teuthology.orchestra.run.smithi059.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-19T01:49:11.314 INFO:teuthology.orchestra.run.smithi059.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-19T01:49:11.314 INFO:teuthology.orchestra.run.smithi059.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-19T01:49:11.315 INFO:teuthology.orchestra.run.smithi059.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-19T01:49:11.318 INFO:teuthology.orchestra.run.smithi059.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-19T01:49:11.322 INFO:teuthology.orchestra.run.smithi059.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-19T01:49:11.487 INFO:teuthology.orchestra.run.smithi059.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-19T01:49:11.569 INFO:teuthology.orchestra.run.smithi059.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-19T01:49:11.685 INFO:teuthology.orchestra.run.smithi059.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-19T01:49:11.735 INFO:teuthology.orchestra.run.smithi059.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-19T01:49:11.769 INFO:teuthology.orchestra.run.smithi059.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-19T01:49:15.282 INFO:teuthology.orchestra.run.smithi081.stdout:Fetched 13.0 MB in 2s (7,118 kB/s) 2024-04-19T01:49:15.336 INFO:teuthology.orchestra.run.smithi179.stdout:Fetched 13.0 MB in 2s (7,167 kB/s) 2024-04-19T01:49:16.384 INFO:teuthology.orchestra.run.smithi059.stdout:Fetched 13.0 MB in 3s (4,884 kB/s) 2024-04-19T01:49:16.695 INFO:teuthology.orchestra.run.smithi081.stdout:Reading package lists... 2024-04-19T01:49:16.713 DEBUG:teuthology.orchestra.run.smithi081:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-19T01:49:16.722 INFO:teuthology.orchestra.run.smithi179.stdout:Reading package lists... 2024-04-19T01:49:16.740 DEBUG:teuthology.orchestra.run.smithi179:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-19T01:49:16.775 INFO:teuthology.orchestra.run.smithi081.stdout:Reading package lists... 2024-04-19T01:49:16.792 INFO:teuthology.orchestra.run.smithi179.stdout:Reading package lists... 2024-04-19T01:49:16.976 INFO:teuthology.orchestra.run.smithi081.stdout:Building dependency tree... 2024-04-19T01:49:16.976 INFO:teuthology.orchestra.run.smithi081.stdout:Reading state information... 2024-04-19T01:49:16.977 INFO:teuthology.orchestra.run.smithi179.stdout:Building dependency tree... 2024-04-19T01:49:16.978 INFO:teuthology.orchestra.run.smithi179.stdout:Reading state information... 2024-04-19T01:49:17.135 INFO:teuthology.orchestra.run.smithi179.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T01:49:17.135 INFO:teuthology.orchestra.run.smithi179.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T01:49:17.135 INFO:teuthology.orchestra.run.smithi179.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T01:49:17.136 INFO:teuthology.orchestra.run.smithi179.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T01:49:17.136 INFO:teuthology.orchestra.run.smithi179.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-19T01:49:17.136 INFO:teuthology.orchestra.run.smithi179.stdout: python2.7-minimal 2024-04-19T01:49:17.136 INFO:teuthology.orchestra.run.smithi179.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T01:49:17.137 INFO:teuthology.orchestra.run.smithi179.stdout:The following additional packages will be installed: 2024-04-19T01:49:17.137 INFO:teuthology.orchestra.run.smithi179.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-19T01:49:17.137 INFO:teuthology.orchestra.run.smithi179.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-19T01:49:17.137 INFO:teuthology.orchestra.run.smithi179.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-19T01:49:17.138 INFO:teuthology.orchestra.run.smithi179.stdout:Suggested packages: 2024-04-19T01:49:17.138 INFO:teuthology.orchestra.run.smithi179.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-19T01:49:17.138 INFO:teuthology.orchestra.run.smithi179.stdout:Recommended packages: 2024-04-19T01:49:17.139 INFO:teuthology.orchestra.run.smithi179.stdout: thermald 2024-04-19T01:49:17.149 INFO:teuthology.orchestra.run.smithi081.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T01:49:17.149 INFO:teuthology.orchestra.run.smithi081.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T01:49:17.149 INFO:teuthology.orchestra.run.smithi081.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T01:49:17.150 INFO:teuthology.orchestra.run.smithi081.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T01:49:17.150 INFO:teuthology.orchestra.run.smithi081.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-19T01:49:17.150 INFO:teuthology.orchestra.run.smithi081.stdout: python2.7-minimal 2024-04-19T01:49:17.150 INFO:teuthology.orchestra.run.smithi081.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T01:49:17.151 INFO:teuthology.orchestra.run.smithi081.stdout:The following additional packages will be installed: 2024-04-19T01:49:17.151 INFO:teuthology.orchestra.run.smithi081.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-19T01:49:17.152 INFO:teuthology.orchestra.run.smithi081.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-19T01:49:17.152 INFO:teuthology.orchestra.run.smithi081.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-19T01:49:17.153 INFO:teuthology.orchestra.run.smithi081.stdout:Suggested packages: 2024-04-19T01:49:17.153 INFO:teuthology.orchestra.run.smithi081.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-19T01:49:17.153 INFO:teuthology.orchestra.run.smithi081.stdout:Recommended packages: 2024-04-19T01:49:17.153 INFO:teuthology.orchestra.run.smithi081.stdout: thermald 2024-04-19T01:49:17.184 INFO:teuthology.orchestra.run.smithi179.stdout:The following NEW packages will be installed: 2024-04-19T01:49:17.184 INFO:teuthology.orchestra.run.smithi179.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-19T01:49:17.184 INFO:teuthology.orchestra.run.smithi179.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-19T01:49:17.184 INFO:teuthology.orchestra.run.smithi179.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-19T01:49:17.186 INFO:teuthology.orchestra.run.smithi179.stdout:The following packages will be upgraded: 2024-04-19T01:49:17.186 INFO:teuthology.orchestra.run.smithi179.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-19T01:49:17.194 INFO:teuthology.orchestra.run.smithi081.stdout:The following NEW packages will be installed: 2024-04-19T01:49:17.195 INFO:teuthology.orchestra.run.smithi081.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-19T01:49:17.195 INFO:teuthology.orchestra.run.smithi081.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-19T01:49:17.195 INFO:teuthology.orchestra.run.smithi081.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-19T01:49:17.196 INFO:teuthology.orchestra.run.smithi081.stdout:The following packages will be upgraded: 2024-04-19T01:49:17.197 INFO:teuthology.orchestra.run.smithi081.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-19T01:49:17.242 INFO:teuthology.orchestra.run.smithi179.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-19T01:49:17.242 INFO:teuthology.orchestra.run.smithi179.stdout:Need to get 113 MB of archives. 2024-04-19T01:49:17.243 INFO:teuthology.orchestra.run.smithi179.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-19T01:49:17.243 INFO:teuthology.orchestra.run.smithi179.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-19T01:49:17.258 INFO:teuthology.orchestra.run.smithi081.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-19T01:49:17.259 INFO:teuthology.orchestra.run.smithi081.stdout:Need to get 113 MB of archives. 2024-04-19T01:49:17.259 INFO:teuthology.orchestra.run.smithi081.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-19T01:49:17.259 INFO:teuthology.orchestra.run.smithi081.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-19T01:49:17.654 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2024-04-19T01:49:17.671 DEBUG:teuthology.orchestra.run.smithi059:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-19T01:49:17.732 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2024-04-19T01:49:17.944 INFO:teuthology.orchestra.run.smithi059.stdout:Building dependency tree... 2024-04-19T01:49:17.944 INFO:teuthology.orchestra.run.smithi059.stdout:Reading state information... 2024-04-19T01:49:17.996 INFO:teuthology.orchestra.run.smithi179.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-19T01:49:18.056 INFO:teuthology.orchestra.run.smithi081.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-19T01:49:18.127 INFO:teuthology.orchestra.run.smithi059.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T01:49:18.127 INFO:teuthology.orchestra.run.smithi059.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T01:49:18.127 INFO:teuthology.orchestra.run.smithi059.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T01:49:18.127 INFO:teuthology.orchestra.run.smithi059.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T01:49:18.127 INFO:teuthology.orchestra.run.smithi059.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-19T01:49:18.128 INFO:teuthology.orchestra.run.smithi059.stdout: python2.7-minimal 2024-04-19T01:49:18.128 INFO:teuthology.orchestra.run.smithi059.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T01:49:18.129 INFO:teuthology.orchestra.run.smithi059.stdout:The following additional packages will be installed: 2024-04-19T01:49:18.129 INFO:teuthology.orchestra.run.smithi059.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-19T01:49:18.129 INFO:teuthology.orchestra.run.smithi059.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-19T01:49:18.129 INFO:teuthology.orchestra.run.smithi059.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-19T01:49:18.130 INFO:teuthology.orchestra.run.smithi059.stdout:Suggested packages: 2024-04-19T01:49:18.131 INFO:teuthology.orchestra.run.smithi059.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-19T01:49:18.131 INFO:teuthology.orchestra.run.smithi059.stdout:Recommended packages: 2024-04-19T01:49:18.131 INFO:teuthology.orchestra.run.smithi059.stdout: thermald 2024-04-19T01:49:18.177 INFO:teuthology.orchestra.run.smithi059.stdout:The following NEW packages will be installed: 2024-04-19T01:49:18.177 INFO:teuthology.orchestra.run.smithi059.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-19T01:49:18.177 INFO:teuthology.orchestra.run.smithi059.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-19T01:49:18.178 INFO:teuthology.orchestra.run.smithi059.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-19T01:49:18.178 INFO:teuthology.orchestra.run.smithi059.stdout:The following packages will be upgraded: 2024-04-19T01:49:18.179 INFO:teuthology.orchestra.run.smithi059.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-19T01:49:18.359 INFO:teuthology.orchestra.run.smithi059.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-19T01:49:18.359 INFO:teuthology.orchestra.run.smithi059.stdout:Need to get 113 MB of archives. 2024-04-19T01:49:18.359 INFO:teuthology.orchestra.run.smithi059.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-19T01:49:18.359 INFO:teuthology.orchestra.run.smithi059.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-19T01:49:18.365 INFO:teuthology.orchestra.run.smithi179.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-19T01:49:18.392 INFO:teuthology.orchestra.run.smithi081.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-19T01:49:20.098 INFO:teuthology.orchestra.run.smithi059.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-19T01:49:20.114 INFO:teuthology.orchestra.run.smithi081.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-19T01:49:20.114 INFO:teuthology.orchestra.run.smithi081.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-19T01:49:20.224 INFO:teuthology.orchestra.run.smithi081.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-19T01:49:20.242 INFO:teuthology.orchestra.run.smithi179.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-19T01:49:20.243 INFO:teuthology.orchestra.run.smithi179.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-19T01:49:20.326 INFO:teuthology.orchestra.run.smithi179.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-19T01:49:20.391 INFO:teuthology.orchestra.run.smithi081.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-19T01:49:20.436 INFO:teuthology.orchestra.run.smithi081.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-19T01:49:20.442 INFO:teuthology.orchestra.run.smithi059.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-19T01:49:20.519 INFO:teuthology.orchestra.run.smithi179.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-19T01:49:20.564 INFO:teuthology.orchestra.run.smithi179.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-19T01:49:20.829 INFO:teuthology.orchestra.run.smithi081.stdout:Fetched 113 MB in 3s (35.0 MB/s) 2024-04-19T01:49:20.973 INFO:teuthology.orchestra.run.smithi179.stdout:Fetched 113 MB in 3s (33.6 MB/s) 2024-04-19T01:49:20.984 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-19T01:49:21.128 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-19T01:49:23.170 INFO:teuthology.orchestra.run.smithi179.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-19T01:49:23.174 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T01:49:23.209 INFO:teuthology.orchestra.run.smithi081.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-19T01:49:23.213 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T01:49:23.250 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:49:23.324 INFO:teuthology.orchestra.run.smithi059.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-19T01:49:23.325 INFO:teuthology.orchestra.run.smithi059.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-19T01:49:23.325 INFO:teuthology.orchestra.run.smithi059.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-19T01:49:23.549 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:49:25.119 INFO:teuthology.orchestra.run.smithi059.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-19T01:49:25.259 INFO:teuthology.orchestra.run.smithi059.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-19T01:49:25.426 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-19T01:49:25.448 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T01:49:25.535 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:49:25.700 INFO:teuthology.orchestra.run.smithi059.stdout:Fetched 113 MB in 7s (16.0 MB/s) 2024-04-19T01:49:25.725 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-19T01:49:25.746 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T01:49:25.837 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:49:25.968 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-19T01:49:25.988 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-19T01:49:26.009 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T01:49:26.037 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:49:26.270 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-19T01:49:26.291 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T01:49:26.328 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:49:27.976 INFO:teuthology.orchestra.run.smithi059.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-19T01:49:27.981 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T01:49:28.164 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:49:30.557 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-19T01:49:30.578 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T01:49:30.665 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:49:31.143 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-19T01:49:31.164 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T01:49:31.201 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:49:32.954 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T01:49:33.097 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T01:49:33.439 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T01:49:33.507 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T01:49:33.568 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T01:49:33.616 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T01:49:33.862 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-19T01:49:33.873 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-19T01:49:33.904 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-19T01:49:33.950 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T01:49:34.078 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T01:49:34.414 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-19T01:49:34.437 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-19T01:49:34.489 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-19T01:49:37.358 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T01:49:37.485 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T01:49:37.826 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T01:49:37.955 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T01:49:38.258 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-19T01:49:38.281 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-19T01:49:38.316 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-19T01:49:40.006 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-19T01:49:40.043 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T01:49:40.080 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:49:40.164 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-19T01:49:40.202 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T01:49:40.240 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:49:42.653 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T01:49:42.775 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T01:49:42.961 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T01:49:43.085 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T01:49:43.116 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-19T01:49:43.238 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:49:43.364 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-19T01:49:43.453 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-19T01:49:43.493 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:49:43.565 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:49:43.707 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-19T01:49:43.765 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-19T01:49:43.804 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T01:49:43.836 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:49:43.840 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:49:44.991 INFO:teuthology.orchestra.run.smithi179.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-19T01:49:44.991 INFO:teuthology.orchestra.run.smithi179.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-19T01:49:44.991 INFO:teuthology.orchestra.run.smithi179.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-19T01:49:44.991 INFO:teuthology.orchestra.run.smithi179.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-19T01:49:45.332 INFO:teuthology.orchestra.run.smithi081.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-19T01:49:45.333 INFO:teuthology.orchestra.run.smithi081.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-19T01:49:45.333 INFO:teuthology.orchestra.run.smithi081.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-19T01:49:45.333 INFO:teuthology.orchestra.run.smithi081.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-19T01:49:45.537 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:49:45.597 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:49:46.981 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:49:47.041 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:49:47.511 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T01:49:47.608 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T01:49:47.924 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-19T01:49:48.037 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:49:48.163 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-19T01:49:48.293 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:49:48.381 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-19T01:49:48.507 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-19T01:49:48.583 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-19T01:49:48.642 INFO:teuthology.orchestra.run.smithi179.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:49:48.700 INFO:teuthology.orchestra.run.smithi179.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-19T01:49:48.701 INFO:teuthology.orchestra.run.smithi179.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-19T01:49:48.709 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-19T01:49:48.835 INFO:teuthology.orchestra.run.smithi081.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:49:48.904 INFO:teuthology.orchestra.run.smithi081.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-19T01:49:48.904 INFO:teuthology.orchestra.run.smithi081.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-19T01:49:49.623 INFO:teuthology.orchestra.run.smithi059.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-19T01:49:49.623 INFO:teuthology.orchestra.run.smithi059.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-19T01:49:49.623 INFO:teuthology.orchestra.run.smithi059.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-19T01:49:49.623 INFO:teuthology.orchestra.run.smithi059.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-19T01:49:49.901 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:49:51.645 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:49:53.079 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-19T01:49:53.204 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-19T01:49:53.331 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:49:53.397 INFO:teuthology.orchestra.run.smithi059.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-19T01:49:53.397 INFO:teuthology.orchestra.run.smithi059.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-19T01:50:03.067 INFO:teuthology.orchestra.run.smithi179.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-19T01:50:03.067 INFO:teuthology.orchestra.run.smithi179.stdout:Sourcing file `/etc/default/grub' 2024-04-19T01:50:03.098 INFO:teuthology.orchestra.run.smithi179.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-19T01:50:03.113 INFO:teuthology.orchestra.run.smithi179.stdout:Generating grub configuration file ... 2024-04-19T01:50:03.184 INFO:teuthology.orchestra.run.smithi081.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-19T01:50:03.185 INFO:teuthology.orchestra.run.smithi081.stdout:Sourcing file `/etc/default/grub' 2024-04-19T01:50:03.207 INFO:teuthology.orchestra.run.smithi081.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-19T01:50:03.219 INFO:teuthology.orchestra.run.smithi081.stdout:Generating grub configuration file ... 2024-04-19T01:50:03.362 INFO:teuthology.orchestra.run.smithi179.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-19T01:50:03.377 INFO:teuthology.orchestra.run.smithi179.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-19T01:50:03.637 INFO:teuthology.orchestra.run.smithi081.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-19T01:50:03.655 INFO:teuthology.orchestra.run.smithi081.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-19T01:50:03.666 INFO:teuthology.orchestra.run.smithi179.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-19T01:50:03.672 INFO:teuthology.orchestra.run.smithi179.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-19T01:50:03.745 INFO:teuthology.orchestra.run.smithi179.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-19T01:50:03.750 INFO:teuthology.orchestra.run.smithi179.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-19T01:50:03.896 INFO:teuthology.orchestra.run.smithi179.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-19T01:50:03.896 INFO:teuthology.orchestra.run.smithi179.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-19T01:50:03.896 INFO:teuthology.orchestra.run.smithi179.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-19T01:50:03.942 INFO:teuthology.orchestra.run.smithi179.stdout:done 2024-04-19T01:50:04.079 INFO:teuthology.orchestra.run.smithi081.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-19T01:50:04.086 INFO:teuthology.orchestra.run.smithi081.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-19T01:50:04.151 INFO:teuthology.orchestra.run.smithi081.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-19T01:50:04.158 INFO:teuthology.orchestra.run.smithi081.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-19T01:50:04.310 INFO:teuthology.orchestra.run.smithi081.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-19T01:50:04.310 INFO:teuthology.orchestra.run.smithi081.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-19T01:50:04.310 INFO:teuthology.orchestra.run.smithi081.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-19T01:50:04.351 INFO:teuthology.orchestra.run.smithi081.stdout:done 2024-04-19T01:50:04.409 DEBUG:teuthology.orchestra.run.smithi179:> dpkg -s linux-image-generic 2024-04-19T01:50:04.431 INFO:teuthology.orchestra.run.smithi179.stdout:Package: linux-image-generic 2024-04-19T01:50:04.431 INFO:teuthology.orchestra.run.smithi179.stdout:Status: install ok installed 2024-04-19T01:50:04.431 INFO:teuthology.orchestra.run.smithi179.stdout:Priority: optional 2024-04-19T01:50:04.431 INFO:teuthology.orchestra.run.smithi179.stdout:Section: kernel 2024-04-19T01:50:04.431 INFO:teuthology.orchestra.run.smithi179.stdout:Installed-Size: 21 2024-04-19T01:50:04.431 INFO:teuthology.orchestra.run.smithi179.stdout:Maintainer: Ubuntu Kernel Team 2024-04-19T01:50:04.432 INFO:teuthology.orchestra.run.smithi179.stdout:Architecture: amd64 2024-04-19T01:50:04.432 INFO:teuthology.orchestra.run.smithi179.stdout:Source: linux-meta 2024-04-19T01:50:04.432 INFO:teuthology.orchestra.run.smithi179.stdout:Version: 5.15.0.105.102 2024-04-19T01:50:04.432 INFO:teuthology.orchestra.run.smithi179.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-19T01:50:04.432 INFO:teuthology.orchestra.run.smithi179.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-19T01:50:04.432 INFO:teuthology.orchestra.run.smithi179.stdout:Recommends: thermald 2024-04-19T01:50:04.432 INFO:teuthology.orchestra.run.smithi179.stdout:Description: Generic Linux kernel image 2024-04-19T01:50:04.432 INFO:teuthology.orchestra.run.smithi179.stdout: This package will always depend on the latest generic kernel image 2024-04-19T01:50:04.432 INFO:teuthology.orchestra.run.smithi179.stdout: available. 2024-04-19T01:50:04.432 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-19T01:50:04.432 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-19T01:50:04.433 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-19T01:50:04.433 INFO:teuthology.task.kernel:Installing distro kernel on osd.8... 2024-04-19T01:50:04.433 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi179.front.sepia.ceph.com, path=None, version=distro) 2024-04-19T01:50:04.433 DEBUG:teuthology.orchestra.run.smithi179:> sudo apt-get clean 2024-04-19T01:50:04.584 DEBUG:teuthology.orchestra.run.smithi179:> sudo apt-get update 2024-04-19T01:50:04.699 INFO:teuthology.orchestra.run.smithi179.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T01:50:04.708 INFO:teuthology.orchestra.run.smithi179.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-19T01:50:04.728 INFO:teuthology.orchestra.run.smithi179.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-19T01:50:04.812 INFO:teuthology.orchestra.run.smithi179.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-19T01:50:04.843 DEBUG:teuthology.orchestra.run.smithi081:> dpkg -s linux-image-generic 2024-04-19T01:50:04.860 INFO:teuthology.orchestra.run.smithi081.stdout:Package: linux-image-generic 2024-04-19T01:50:04.861 INFO:teuthology.orchestra.run.smithi081.stdout:Status: install ok installed 2024-04-19T01:50:04.861 INFO:teuthology.orchestra.run.smithi081.stdout:Priority: optional 2024-04-19T01:50:04.861 INFO:teuthology.orchestra.run.smithi081.stdout:Section: kernel 2024-04-19T01:50:04.861 INFO:teuthology.orchestra.run.smithi081.stdout:Installed-Size: 21 2024-04-19T01:50:04.861 INFO:teuthology.orchestra.run.smithi081.stdout:Maintainer: Ubuntu Kernel Team 2024-04-19T01:50:04.861 INFO:teuthology.orchestra.run.smithi081.stdout:Architecture: amd64 2024-04-19T01:50:04.861 INFO:teuthology.orchestra.run.smithi081.stdout:Source: linux-meta 2024-04-19T01:50:04.861 INFO:teuthology.orchestra.run.smithi081.stdout:Version: 5.15.0.105.102 2024-04-19T01:50:04.861 INFO:teuthology.orchestra.run.smithi081.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-19T01:50:04.861 INFO:teuthology.orchestra.run.smithi081.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-19T01:50:04.861 INFO:teuthology.orchestra.run.smithi081.stdout:Recommends: thermald 2024-04-19T01:50:04.861 INFO:teuthology.orchestra.run.smithi081.stdout:Description: Generic Linux kernel image 2024-04-19T01:50:04.861 INFO:teuthology.orchestra.run.smithi081.stdout: This package will always depend on the latest generic kernel image 2024-04-19T01:50:04.862 INFO:teuthology.orchestra.run.smithi081.stdout: available. 2024-04-19T01:50:04.862 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-19T01:50:04.862 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-19T01:50:04.862 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-19T01:50:04.862 INFO:teuthology.task.kernel:Installing distro kernel on osd.4... 2024-04-19T01:50:04.862 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi081.front.sepia.ceph.com, path=None, version=distro) 2024-04-19T01:50:04.862 DEBUG:teuthology.orchestra.run.smithi081:> sudo apt-get clean 2024-04-19T01:50:05.082 DEBUG:teuthology.orchestra.run.smithi081:> sudo apt-get update 2024-04-19T01:50:05.197 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-19T01:50:05.314 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T01:50:05.414 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-19T01:50:05.477 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-19T01:50:05.939 INFO:teuthology.orchestra.run.smithi179.stdout:Reading package lists... 2024-04-19T01:50:05.957 DEBUG:teuthology.orchestra.run.smithi179:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-19T01:50:06.007 INFO:teuthology.orchestra.run.smithi179.stdout:Reading package lists... 2024-04-19T01:50:06.193 INFO:teuthology.orchestra.run.smithi179.stdout:Building dependency tree... 2024-04-19T01:50:06.193 INFO:teuthology.orchestra.run.smithi179.stdout:Reading state information... 2024-04-19T01:50:06.353 INFO:teuthology.orchestra.run.smithi179.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-19T01:50:06.353 INFO:teuthology.orchestra.run.smithi179.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T01:50:06.354 INFO:teuthology.orchestra.run.smithi179.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T01:50:06.354 INFO:teuthology.orchestra.run.smithi179.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T01:50:06.354 INFO:teuthology.orchestra.run.smithi179.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T01:50:06.354 INFO:teuthology.orchestra.run.smithi179.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-19T01:50:06.354 INFO:teuthology.orchestra.run.smithi179.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-19T01:50:06.354 INFO:teuthology.orchestra.run.smithi179.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T01:50:06.389 INFO:teuthology.orchestra.run.smithi179.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-19T01:50:06.391 DEBUG:teuthology.orchestra.run.smithi179:> dpkg -s linux-image-generic 2024-04-19T01:50:06.453 INFO:teuthology.orchestra.run.smithi179.stdout:Package: linux-image-generic 2024-04-19T01:50:06.453 INFO:teuthology.orchestra.run.smithi179.stdout:Status: install ok installed 2024-04-19T01:50:06.453 INFO:teuthology.orchestra.run.smithi179.stdout:Priority: optional 2024-04-19T01:50:06.453 INFO:teuthology.orchestra.run.smithi179.stdout:Section: kernel 2024-04-19T01:50:06.453 INFO:teuthology.orchestra.run.smithi179.stdout:Installed-Size: 21 2024-04-19T01:50:06.453 INFO:teuthology.orchestra.run.smithi179.stdout:Maintainer: Ubuntu Kernel Team 2024-04-19T01:50:06.453 INFO:teuthology.orchestra.run.smithi179.stdout:Architecture: amd64 2024-04-19T01:50:06.454 INFO:teuthology.orchestra.run.smithi179.stdout:Source: linux-meta 2024-04-19T01:50:06.454 INFO:teuthology.orchestra.run.smithi179.stdout:Version: 5.15.0.105.102 2024-04-19T01:50:06.454 INFO:teuthology.orchestra.run.smithi179.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-19T01:50:06.454 INFO:teuthology.orchestra.run.smithi179.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-19T01:50:06.454 INFO:teuthology.orchestra.run.smithi179.stdout:Recommends: thermald 2024-04-19T01:50:06.454 INFO:teuthology.orchestra.run.smithi179.stdout:Description: Generic Linux kernel image 2024-04-19T01:50:06.454 INFO:teuthology.orchestra.run.smithi179.stdout: This package will always depend on the latest generic kernel image 2024-04-19T01:50:06.454 INFO:teuthology.orchestra.run.smithi179.stdout: available. 2024-04-19T01:50:06.454 DEBUG:teuthology.orchestra.run.smithi179:> mktemp 2024-04-19T01:50:06.498 INFO:teuthology.orchestra.run.smithi179.stdout:/tmp/tmp.50QvHOopEk 2024-04-19T01:50:06.499 DEBUG:teuthology.orchestra.run.smithi179:> sudo cp /boot/grub/grub.cfg /tmp/tmp.50QvHOopEk 2024-04-19T01:50:06.556 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 0666 /tmp/tmp.50QvHOopEk 2024-04-19T01:50:06.586 INFO:teuthology.orchestra.run.smithi081.stdout:Reading package lists... 2024-04-19T01:50:06.605 DEBUG:teuthology.orchestra.run.smithi081:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-19T01:50:06.656 INFO:teuthology.orchestra.run.smithi081.stdout:Reading package lists... 2024-04-19T01:50:06.672 DEBUG:teuthology.orchestra.remote:smithi179:/tmp/tmp.50QvHOopEk is 10KB 2024-04-19T01:50:06.689 DEBUG:teuthology.orchestra.run.smithi179:> rm -fr /tmp/tmp.50QvHOopEk 2024-04-19T01:50:06.694 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-19T01:50:06.755 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-04-19T01:50:06.755 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-19T01:50:06.755 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-19T01:50:06.831 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-19T01:50:06.831 DEBUG:teuthology.orchestra.run.smithi179:> sudo update-grub 2024-04-19T01:50:06.833 INFO:teuthology.orchestra.run.smithi081.stdout:Building dependency tree... 2024-04-19T01:50:06.834 INFO:teuthology.orchestra.run.smithi081.stdout:Reading state information... 2024-04-19T01:50:06.987 INFO:teuthology.orchestra.run.smithi081.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-19T01:50:06.987 INFO:teuthology.orchestra.run.smithi081.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T01:50:06.987 INFO:teuthology.orchestra.run.smithi081.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T01:50:06.987 INFO:teuthology.orchestra.run.smithi081.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T01:50:06.987 INFO:teuthology.orchestra.run.smithi081.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T01:50:06.987 INFO:teuthology.orchestra.run.smithi081.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-19T01:50:06.988 INFO:teuthology.orchestra.run.smithi081.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-19T01:50:06.988 INFO:teuthology.orchestra.run.smithi081.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T01:50:07.027 INFO:teuthology.orchestra.run.smithi081.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-19T01:50:07.029 DEBUG:teuthology.orchestra.run.smithi081:> dpkg -s linux-image-generic 2024-04-19T01:50:07.083 INFO:teuthology.orchestra.run.smithi081.stdout:Package: linux-image-generic 2024-04-19T01:50:07.083 INFO:teuthology.orchestra.run.smithi081.stdout:Status: install ok installed 2024-04-19T01:50:07.083 INFO:teuthology.orchestra.run.smithi081.stdout:Priority: optional 2024-04-19T01:50:07.083 INFO:teuthology.orchestra.run.smithi081.stdout:Section: kernel 2024-04-19T01:50:07.083 INFO:teuthology.orchestra.run.smithi081.stdout:Installed-Size: 21 2024-04-19T01:50:07.083 INFO:teuthology.orchestra.run.smithi081.stdout:Maintainer: Ubuntu Kernel Team 2024-04-19T01:50:07.084 INFO:teuthology.orchestra.run.smithi081.stdout:Architecture: amd64 2024-04-19T01:50:07.084 INFO:teuthology.orchestra.run.smithi081.stdout:Source: linux-meta 2024-04-19T01:50:07.084 INFO:teuthology.orchestra.run.smithi081.stdout:Version: 5.15.0.105.102 2024-04-19T01:50:07.084 INFO:teuthology.orchestra.run.smithi081.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-19T01:50:07.084 INFO:teuthology.orchestra.run.smithi081.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-19T01:50:07.084 INFO:teuthology.orchestra.run.smithi081.stdout:Recommends: thermald 2024-04-19T01:50:07.084 INFO:teuthology.orchestra.run.smithi081.stdout:Description: Generic Linux kernel image 2024-04-19T01:50:07.084 INFO:teuthology.orchestra.run.smithi081.stdout: This package will always depend on the latest generic kernel image 2024-04-19T01:50:07.084 INFO:teuthology.orchestra.run.smithi081.stdout: available. 2024-04-19T01:50:07.084 DEBUG:teuthology.orchestra.run.smithi081:> mktemp 2024-04-19T01:50:07.131 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.60SohcSdSY 2024-04-19T01:50:07.131 DEBUG:teuthology.orchestra.run.smithi081:> sudo cp /boot/grub/grub.cfg /tmp/tmp.60SohcSdSY 2024-04-19T01:50:07.187 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.60SohcSdSY 2024-04-19T01:50:07.301 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.60SohcSdSY is 10KB 2024-04-19T01:50:07.314 DEBUG:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.60SohcSdSY 2024-04-19T01:50:07.319 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-19T01:50:07.375 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-19T01:50:07.375 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-19T01:50:07.375 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-19T01:50:07.465 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-19T01:50:07.465 DEBUG:teuthology.orchestra.run.smithi081:> sudo update-grub 2024-04-19T01:50:07.707 INFO:teuthology.orchestra.run.smithi059.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-19T01:50:07.707 INFO:teuthology.orchestra.run.smithi059.stdout:Sourcing file `/etc/default/grub' 2024-04-19T01:50:07.719 INFO:teuthology.orchestra.run.smithi059.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-19T01:50:07.745 INFO:teuthology.orchestra.run.smithi059.stdout:Generating grub configuration file ... 2024-04-19T01:50:08.009 INFO:teuthology.orchestra.run.smithi059.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-19T01:50:08.032 INFO:teuthology.orchestra.run.smithi059.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-19T01:50:08.088 INFO:teuthology.orchestra.run.smithi179.stderr:Sourcing file `/etc/default/grub' 2024-04-19T01:50:08.089 INFO:teuthology.orchestra.run.smithi179.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-19T01:50:08.091 INFO:teuthology.orchestra.run.smithi179.stderr:Generating grub configuration file ... 2024-04-19T01:50:08.270 INFO:teuthology.orchestra.run.smithi179.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-19T01:50:08.275 INFO:teuthology.orchestra.run.smithi179.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-19T01:50:08.350 INFO:teuthology.orchestra.run.smithi059.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-19T01:50:08.355 INFO:teuthology.orchestra.run.smithi059.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-19T01:50:08.430 INFO:teuthology.orchestra.run.smithi059.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-19T01:50:08.436 INFO:teuthology.orchestra.run.smithi059.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-19T01:50:08.518 INFO:teuthology.orchestra.run.smithi179.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-19T01:50:08.523 INFO:teuthology.orchestra.run.smithi179.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-19T01:50:08.561 INFO:teuthology.orchestra.run.smithi179.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-19T01:50:08.565 INFO:teuthology.orchestra.run.smithi179.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-19T01:50:08.610 INFO:teuthology.orchestra.run.smithi059.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-19T01:50:08.611 INFO:teuthology.orchestra.run.smithi059.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-19T01:50:08.611 INFO:teuthology.orchestra.run.smithi059.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-19T01:50:08.650 INFO:teuthology.orchestra.run.smithi059.stdout:done 2024-04-19T01:50:08.714 INFO:teuthology.orchestra.run.smithi179.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-19T01:50:08.752 INFO:teuthology.orchestra.run.smithi179.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-19T01:50:08.753 INFO:teuthology.orchestra.run.smithi179.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-19T01:50:08.755 INFO:teuthology.orchestra.run.smithi081.stderr:Sourcing file `/etc/default/grub' 2024-04-19T01:50:08.755 INFO:teuthology.orchestra.run.smithi081.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-19T01:50:08.755 INFO:teuthology.orchestra.run.smithi081.stderr:Generating grub configuration file ... 2024-04-19T01:50:08.755 INFO:teuthology.orchestra.run.smithi179.stderr:done 2024-04-19T01:50:08.755 DEBUG:teuthology.orchestra.run.smithi179:> sudo shutdown -r now 2024-04-19T01:50:08.892 INFO:teuthology.orchestra.run.smithi081.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-19T01:50:08.898 INFO:teuthology.orchestra.run.smithi081.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-19T01:50:09.082 DEBUG:teuthology.orchestra.run.smithi059:> dpkg -s linux-image-generic 2024-04-19T01:50:09.110 INFO:teuthology.orchestra.run.smithi059.stdout:Package: linux-image-generic 2024-04-19T01:50:09.110 INFO:teuthology.orchestra.run.smithi059.stdout:Status: install ok installed 2024-04-19T01:50:09.110 INFO:teuthology.orchestra.run.smithi059.stdout:Priority: optional 2024-04-19T01:50:09.110 INFO:teuthology.orchestra.run.smithi059.stdout:Section: kernel 2024-04-19T01:50:09.110 INFO:teuthology.orchestra.run.smithi059.stdout:Installed-Size: 21 2024-04-19T01:50:09.110 INFO:teuthology.orchestra.run.smithi059.stdout:Maintainer: Ubuntu Kernel Team 2024-04-19T01:50:09.110 INFO:teuthology.orchestra.run.smithi059.stdout:Architecture: amd64 2024-04-19T01:50:09.110 INFO:teuthology.orchestra.run.smithi059.stdout:Source: linux-meta 2024-04-19T01:50:09.110 INFO:teuthology.orchestra.run.smithi059.stdout:Version: 5.15.0.105.102 2024-04-19T01:50:09.110 INFO:teuthology.orchestra.run.smithi059.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-19T01:50:09.110 INFO:teuthology.orchestra.run.smithi059.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-19T01:50:09.111 INFO:teuthology.orchestra.run.smithi059.stdout:Recommends: thermald 2024-04-19T01:50:09.111 INFO:teuthology.orchestra.run.smithi059.stdout:Description: Generic Linux kernel image 2024-04-19T01:50:09.111 INFO:teuthology.orchestra.run.smithi059.stdout: This package will always depend on the latest generic kernel image 2024-04-19T01:50:09.111 INFO:teuthology.orchestra.run.smithi059.stdout: available. 2024-04-19T01:50:09.111 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-19T01:50:09.111 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-19T01:50:09.111 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-19T01:50:09.112 INFO:teuthology.task.kernel:Installing distro kernel on osd.0... 2024-04-19T01:50:09.112 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi059.front.sepia.ceph.com, path=None, version=distro) 2024-04-19T01:50:09.112 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt-get clean 2024-04-19T01:50:09.187 INFO:teuthology.orchestra.run.smithi081.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-19T01:50:09.192 INFO:teuthology.orchestra.run.smithi081.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-19T01:50:09.234 INFO:teuthology.orchestra.run.smithi081.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-19T01:50:09.240 INFO:teuthology.orchestra.run.smithi081.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-19T01:50:09.259 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt-get update 2024-04-19T01:50:09.357 INFO:teuthology.orchestra.run.smithi081.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-19T01:50:09.358 INFO:teuthology.orchestra.run.smithi081.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-19T01:50:09.358 INFO:teuthology.orchestra.run.smithi081.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-19T01:50:09.374 INFO:teuthology.orchestra.run.smithi081.stderr:done 2024-04-19T01:50:09.382 DEBUG:teuthology.orchestra.run.smithi081:> sudo shutdown -r now 2024-04-19T01:50:09.401 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-19T01:50:09.511 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T01:50:09.598 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-19T01:50:09.679 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-19T01:50:10.839 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2024-04-19T01:50:10.857 DEBUG:teuthology.orchestra.run.smithi059:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-19T01:50:10.910 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2024-04-19T01:50:11.092 INFO:teuthology.orchestra.run.smithi059.stdout:Building dependency tree... 2024-04-19T01:50:11.093 INFO:teuthology.orchestra.run.smithi059.stdout:Reading state information... 2024-04-19T01:50:11.254 INFO:teuthology.orchestra.run.smithi059.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-19T01:50:11.254 INFO:teuthology.orchestra.run.smithi059.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T01:50:11.254 INFO:teuthology.orchestra.run.smithi059.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T01:50:11.255 INFO:teuthology.orchestra.run.smithi059.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T01:50:11.255 INFO:teuthology.orchestra.run.smithi059.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T01:50:11.255 INFO:teuthology.orchestra.run.smithi059.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-19T01:50:11.255 INFO:teuthology.orchestra.run.smithi059.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-19T01:50:11.255 INFO:teuthology.orchestra.run.smithi059.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T01:50:11.295 INFO:teuthology.orchestra.run.smithi059.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-19T01:50:11.297 DEBUG:teuthology.orchestra.run.smithi059:> dpkg -s linux-image-generic 2024-04-19T01:50:11.355 INFO:teuthology.orchestra.run.smithi059.stdout:Package: linux-image-generic 2024-04-19T01:50:11.355 INFO:teuthology.orchestra.run.smithi059.stdout:Status: install ok installed 2024-04-19T01:50:11.355 INFO:teuthology.orchestra.run.smithi059.stdout:Priority: optional 2024-04-19T01:50:11.355 INFO:teuthology.orchestra.run.smithi059.stdout:Section: kernel 2024-04-19T01:50:11.355 INFO:teuthology.orchestra.run.smithi059.stdout:Installed-Size: 21 2024-04-19T01:50:11.355 INFO:teuthology.orchestra.run.smithi059.stdout:Maintainer: Ubuntu Kernel Team 2024-04-19T01:50:11.355 INFO:teuthology.orchestra.run.smithi059.stdout:Architecture: amd64 2024-04-19T01:50:11.355 INFO:teuthology.orchestra.run.smithi059.stdout:Source: linux-meta 2024-04-19T01:50:11.355 INFO:teuthology.orchestra.run.smithi059.stdout:Version: 5.15.0.105.102 2024-04-19T01:50:11.355 INFO:teuthology.orchestra.run.smithi059.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-19T01:50:11.356 INFO:teuthology.orchestra.run.smithi059.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-19T01:50:11.356 INFO:teuthology.orchestra.run.smithi059.stdout:Recommends: thermald 2024-04-19T01:50:11.356 INFO:teuthology.orchestra.run.smithi059.stdout:Description: Generic Linux kernel image 2024-04-19T01:50:11.356 INFO:teuthology.orchestra.run.smithi059.stdout: This package will always depend on the latest generic kernel image 2024-04-19T01:50:11.356 INFO:teuthology.orchestra.run.smithi059.stdout: available. 2024-04-19T01:50:11.356 DEBUG:teuthology.orchestra.run.smithi059:> mktemp 2024-04-19T01:50:11.409 INFO:teuthology.orchestra.run.smithi059.stdout:/tmp/tmp.FYhHVhJIKk 2024-04-19T01:50:11.409 DEBUG:teuthology.orchestra.run.smithi059:> sudo cp /boot/grub/grub.cfg /tmp/tmp.FYhHVhJIKk 2024-04-19T01:50:11.464 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0666 /tmp/tmp.FYhHVhJIKk 2024-04-19T01:50:11.587 DEBUG:teuthology.orchestra.remote:smithi059:/tmp/tmp.FYhHVhJIKk is 10KB 2024-04-19T01:50:11.605 DEBUG:teuthology.orchestra.run.smithi059:> rm -fr /tmp/tmp.FYhHVhJIKk 2024-04-19T01:50:11.610 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-19T01:50:11.667 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-19T01:50:11.667 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-19T01:50:11.667 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-19T01:50:11.735 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-19T01:50:11.735 DEBUG:teuthology.orchestra.run.smithi059:> sudo update-grub 2024-04-19T01:50:13.135 INFO:teuthology.orchestra.run.smithi059.stderr:Sourcing file `/etc/default/grub' 2024-04-19T01:50:13.136 INFO:teuthology.orchestra.run.smithi059.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-19T01:50:13.139 INFO:teuthology.orchestra.run.smithi059.stderr:Generating grub configuration file ... 2024-04-19T01:50:13.301 INFO:teuthology.orchestra.run.smithi059.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-19T01:50:13.307 INFO:teuthology.orchestra.run.smithi059.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-19T01:50:13.522 INFO:teuthology.orchestra.run.smithi059.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-19T01:50:13.528 INFO:teuthology.orchestra.run.smithi059.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-19T01:50:13.571 INFO:teuthology.orchestra.run.smithi059.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-19T01:50:13.578 INFO:teuthology.orchestra.run.smithi059.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-19T01:50:13.697 INFO:teuthology.orchestra.run.smithi059.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-19T01:50:13.697 INFO:teuthology.orchestra.run.smithi059.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-19T01:50:13.697 INFO:teuthology.orchestra.run.smithi059.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-19T01:50:13.714 INFO:teuthology.orchestra.run.smithi059.stderr:done 2024-04-19T01:50:13.723 DEBUG:teuthology.orchestra.run.smithi059:> sudo shutdown -r now 2024-04-19T01:50:38.782 INFO:teuthology.task.kernel:Checking client osd.8 for new kernel version... 2024-04-19T01:50:38.783 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-04-19T01:50:38.784 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:50:39.386 INFO:teuthology.task.kernel:Checking client osd.4 for new kernel version... 2024-04-19T01:50:39.386 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-04-19T01:50:39.387 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:50:43.731 INFO:teuthology.task.kernel:Checking client osd.0 for new kernel version... 2024-04-19T01:50:43.731 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-04-19T01:50:43.732 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:50:57.318 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.179 2024-04-19T01:50:57.822 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2024-04-19T01:51:02.174 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.59 2024-04-19T01:51:06.324 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-04-19T01:51:06.325 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:51:06.824 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-04-19T01:51:06.825 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:51:09.406 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.179 2024-04-19T01:51:09.890 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2024-04-19T01:51:11.177 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-04-19T01:51:11.177 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:51:14.242 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.59 2024-04-19T01:51:21.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-04-19T01:51:21.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:51:21.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-04-19T01:51:21.892 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:51:26.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-04-19T01:51:26.247 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:51:39.806 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.179 2024-04-19T01:51:40.318 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2024-04-19T01:51:54.822 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-04-19T01:51:54.824 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:51:55.239 DEBUG:teuthology.orchestra.run.smithi179:> true 2024-04-19T01:51:55.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-04-19T01:51:55.320 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:51:55.519 DEBUG:teuthology.orchestra.run.smithi081:> true 2024-04-19T01:51:56.078 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-04-19T01:51:56.078 INFO:teuthology.task.kernel:Checking kernel version of osd.4, want "5.15.0-105-generic"... 2024-04-19T01:51:56.079 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2024-04-19T01:51:56.125 INFO:teuthology.orchestra.run.smithi081.stdout:5.15.0-105-generic 2024-04-19T01:51:56.125 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-19T01:51:56.125 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-19T01:51:56.125 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-19T01:51:56.196 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-04-19T01:51:56.197 INFO:teuthology.task.kernel:Checking kernel version of osd.8, want "5.15.0-105-generic"... 2024-04-19T01:51:56.197 DEBUG:teuthology.orchestra.run.smithi179:> uname -r 2024-04-19T01:51:56.244 INFO:teuthology.orchestra.run.smithi179.stdout:5.15.0-105-generic 2024-04-19T01:51:56.262 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-19T01:51:56.262 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-19T01:51:56.262 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-19T01:51:57.126 INFO:teuthology.task.kernel:Enabling kdb on osd.4... 2024-04-19T01:51:57.126 DEBUG:teuthology.orchestra.run.smithi081:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-19T01:51:57.195 INFO:teuthology.orchestra.run.smithi081.stdout:ttyS1 2024-04-19T01:51:57.226 DEBUG:teuthology.parallel:result is None 2024-04-19T01:51:57.263 INFO:teuthology.task.kernel:Enabling kdb on osd.8... 2024-04-19T01:51:57.263 DEBUG:teuthology.orchestra.run.smithi179:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-19T01:51:57.353 INFO:teuthology.orchestra.run.smithi179.stdout:ttyS1 2024-04-19T01:51:57.386 DEBUG:teuthology.parallel:result is None 2024-04-19T01:52:26.249 DEBUG:teuthology.orchestra.remote:timed out 2024-04-19T01:52:41.250 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-04-19T01:52:41.251 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:52:41.505 DEBUG:teuthology.orchestra.run.smithi059:> true 2024-04-19T01:52:42.138 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-04-19T01:52:42.139 INFO:teuthology.task.kernel:Checking kernel version of osd.0, want "5.15.0-105-generic"... 2024-04-19T01:52:42.139 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2024-04-19T01:52:42.185 INFO:teuthology.orchestra.run.smithi059.stdout:5.15.0-105-generic 2024-04-19T01:52:42.185 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-19T01:52:42.185 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-19T01:52:42.185 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-19T01:52:43.186 INFO:teuthology.task.kernel:Enabling kdb on osd.0... 2024-04-19T01:52:43.186 DEBUG:teuthology.orchestra.run.smithi059:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-19T01:52:43.248 INFO:teuthology.orchestra.run.smithi059.stdout:ttyS1 2024-04-19T01:52:43.270 DEBUG:teuthology.parallel:result is None 2024-04-19T01:52:43.270 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-19T01:52:43.277 INFO:teuthology.task.internal:Creating test directory... 2024-04-19T01:52:43.278 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-19T01:52:43.280 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-19T01:52:43.283 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-19T01:52:43.289 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-19T01:52:43.297 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-19T01:52:43.303 INFO:teuthology.task.internal:Creating archive directory... 2024-04-19T01:52:43.303 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-19T01:52:43.329 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-19T01:52:43.332 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-19T01:52:43.422 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-19T01:52:43.429 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-19T01:52:43.429 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-19T01:52:43.432 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-19T01:52:43.434 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-19T01:52:43.449 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-19T01:52:43.450 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-19T01:52:43.457 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-19T01:52:43.458 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-19T01:52:43.459 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-19T01:52:43.466 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-19T01:52:43.467 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-19T01:52:43.476 INFO:teuthology.task.internal:Configuring sudo... 2024-04-19T01:52:43.476 DEBUG:teuthology.orchestra.run.smithi059:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-19T01:52:43.509 DEBUG:teuthology.orchestra.run.smithi081:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-19T01:52:43.512 DEBUG:teuthology.orchestra.run.smithi179:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-19T01:52:43.541 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-19T01:52:43.550 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-19T01:52:43.550 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-19T01:52:43.565 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-19T01:52:43.569 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-19T01:52:43.589 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-19T01:52:43.617 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-19T01:52:43.665 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-19T01:52:43.665 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-19T01:52:43.737 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-19T01:52:43.744 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-19T01:52:43.792 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-19T01:52:43.793 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-19T01:52:43.858 DEBUG:teuthology.orchestra.run.smithi179:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-19T01:52:43.864 DEBUG:teuthology.orchestra.run.smithi179:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-19T01:52:43.913 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-04-19T01:52:43.913 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-19T01:52:43.976 DEBUG:teuthology.orchestra.run.smithi059:> sudo service rsyslog restart 2024-04-19T01:52:43.979 DEBUG:teuthology.orchestra.run.smithi081:> sudo service rsyslog restart 2024-04-19T01:52:43.981 DEBUG:teuthology.orchestra.run.smithi179:> sudo service rsyslog restart 2024-04-19T01:52:44.055 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-19T01:52:44.063 INFO:teuthology.task.internal:Starting timer... 2024-04-19T01:52:44.063 INFO:teuthology.run_tasks:Running task pcp... 2024-04-19T01:52:44.116 INFO:teuthology.run_tasks:Running task selinux... 2024-04-19T01:52:44.124 DEBUG:teuthology.task.selinux:Excluding smithi059: OS 'ubuntu' does not support SELinux 2024-04-19T01:52:44.125 DEBUG:teuthology.task.selinux:Excluding smithi081: OS 'ubuntu' does not support SELinux 2024-04-19T01:52:44.125 DEBUG:teuthology.task.selinux:Excluding smithi179: OS 'ubuntu' does not support SELinux 2024-04-19T01:52:44.125 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-19T01:52:44.125 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-19T01:52:44.125 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-19T01:52:44.125 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-19T01:52:44.134 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-19T01:52:44.153 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-19T01:52:44.154 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi059.front.sepia.ceph.com,smithi081.front.sepia.ceph.com,smithi179.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-19T01:59:50.103 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi059.front.sepia.ceph.com'), Remote(name='ubuntu@smithi081.front.sepia.ceph.com'), Remote(name='ubuntu@smithi179.front.sepia.ceph.com')] 2024-04-19T01:59:50.105 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-04-19T01:59:50.106 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:59:50.192 DEBUG:teuthology.orchestra.run.smithi059:> true 2024-04-19T01:59:50.266 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-04-19T01:59:50.267 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-04-19T01:59:50.267 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:59:50.355 DEBUG:teuthology.orchestra.run.smithi081:> true 2024-04-19T01:59:50.434 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-04-19T01:59:50.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-04-19T01:59:50.435 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:59:50.517 DEBUG:teuthology.orchestra.run.smithi179:> true 2024-04-19T01:59:50.594 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-04-19T01:59:50.594 INFO:teuthology.run_tasks:Running task clock... 2024-04-19T01:59:50.606 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-19T01:59:50.606 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-19T01:59:50.607 DEBUG:teuthology.orchestra.run.smithi059:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-19T01:59:50.609 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-19T01:59:50.609 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-19T01:59:50.612 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-19T01:59:50.612 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-19T01:59:50.639 INFO:teuthology.orchestra.run.smithi059.stdout:19 Apr 01:59:50 ntpd[17082]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-19T01:59:50.639 INFO:teuthology.orchestra.run.smithi059.stdout:19 Apr 01:59:50 ntpd[17082]: Command line: ntpd -gq 2024-04-19T01:59:50.639 INFO:teuthology.orchestra.run.smithi059.stdout:19 Apr 01:59:50 ntpd[17082]: ---------------------------------------------------- 2024-04-19T01:59:50.639 INFO:teuthology.orchestra.run.smithi059.stdout:19 Apr 01:59:50 ntpd[17082]: ntp-4 is maintained by Network Time Foundation, 2024-04-19T01:59:50.639 INFO:teuthology.orchestra.run.smithi059.stdout:19 Apr 01:59:50 ntpd[17082]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-19T01:59:50.639 INFO:teuthology.orchestra.run.smithi059.stdout:19 Apr 01:59:50 ntpd[17082]: corporation. Support and training for ntp-4 are 2024-04-19T01:59:50.639 INFO:teuthology.orchestra.run.smithi059.stdout:19 Apr 01:59:50 ntpd[17082]: available at https://www.nwtime.org/support 2024-04-19T01:59:50.639 INFO:teuthology.orchestra.run.smithi059.stdout:19 Apr 01:59:50 ntpd[17082]: ---------------------------------------------------- 2024-04-19T01:59:50.639 INFO:teuthology.orchestra.run.smithi059.stdout:19 Apr 01:59:50 ntpd[17082]: proto: precision = 0.071 usec (-24) 2024-04-19T01:59:50.639 INFO:teuthology.orchestra.run.smithi059.stdout:19 Apr 01:59:50 ntpd[17082]: basedate set to 2022-02-04 2024-04-19T01:59:50.639 INFO:teuthology.orchestra.run.smithi059.stdout:19 Apr 01:59:50 ntpd[17082]: gps base set to 2022-02-06 (week 2196) 2024-04-19T01:59:50.639 INFO:teuthology.orchestra.run.smithi059.stdout:19 Apr 01:59:50 ntpd[17082]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-19T01:59:50.640 INFO:teuthology.orchestra.run.smithi059.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-19T01:59:50.640 INFO:teuthology.orchestra.run.smithi059.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-19T01:59:50.641 INFO:teuthology.orchestra.run.smithi059.stdout:19 Apr 01:59:50 ntpd[17082]: restrict ::: KOD does nothing without LIMITED. 2024-04-19T01:59:50.641 INFO:teuthology.orchestra.run.smithi059.stdout:19 Apr 01:59:50 ntpd[17082]: Listen and drop on 0 v6wildcard [::]:123 2024-04-19T01:59:50.641 INFO:teuthology.orchestra.run.smithi059.stdout:19 Apr 01:59:50 ntpd[17082]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-19T01:59:50.641 INFO:teuthology.orchestra.run.smithi059.stdout:19 Apr 01:59:50 ntpd[17082]: Listen normally on 2 lo 127.0.0.1:123 2024-04-19T01:59:50.641 INFO:teuthology.orchestra.run.smithi059.stdout:19 Apr 01:59:50 ntpd[17082]: Listen normally on 3 ens1f0 172.21.15.59:123 2024-04-19T01:59:50.641 INFO:teuthology.orchestra.run.smithi059.stdout:19 Apr 01:59:50 ntpd[17082]: Listen normally on 4 lo [::1]:123 2024-04-19T01:59:50.641 INFO:teuthology.orchestra.run.smithi059.stdout:19 Apr 01:59:50 ntpd[17082]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:192a%4]:123 2024-04-19T01:59:50.641 INFO:teuthology.orchestra.run.smithi059.stdout:19 Apr 01:59:50 ntpd[17082]: Listening on routing socket on fd #22 for interface updates 2024-04-19T01:59:50.642 INFO:teuthology.orchestra.run.smithi081.stdout:19 Apr 01:59:50 ntpd[17209]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-19T01:59:50.642 INFO:teuthology.orchestra.run.smithi081.stdout:19 Apr 01:59:50 ntpd[17209]: Command line: ntpd -gq 2024-04-19T01:59:50.642 INFO:teuthology.orchestra.run.smithi081.stdout:19 Apr 01:59:50 ntpd[17209]: ---------------------------------------------------- 2024-04-19T01:59:50.642 INFO:teuthology.orchestra.run.smithi081.stdout:19 Apr 01:59:50 ntpd[17209]: ntp-4 is maintained by Network Time Foundation, 2024-04-19T01:59:50.642 INFO:teuthology.orchestra.run.smithi081.stdout:19 Apr 01:59:50 ntpd[17209]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-19T01:59:50.642 INFO:teuthology.orchestra.run.smithi081.stdout:19 Apr 01:59:50 ntpd[17209]: corporation. Support and training for ntp-4 are 2024-04-19T01:59:50.643 INFO:teuthology.orchestra.run.smithi081.stdout:19 Apr 01:59:50 ntpd[17209]: available at https://www.nwtime.org/support 2024-04-19T01:59:50.643 INFO:teuthology.orchestra.run.smithi081.stdout:19 Apr 01:59:50 ntpd[17209]: ---------------------------------------------------- 2024-04-19T01:59:50.643 INFO:teuthology.orchestra.run.smithi081.stdout:19 Apr 01:59:50 ntpd[17209]: proto: precision = 0.059 usec (-24) 2024-04-19T01:59:50.643 INFO:teuthology.orchestra.run.smithi081.stdout:19 Apr 01:59:50 ntpd[17209]: basedate set to 2022-02-04 2024-04-19T01:59:50.643 INFO:teuthology.orchestra.run.smithi081.stdout:19 Apr 01:59:50 ntpd[17209]: gps base set to 2022-02-06 (week 2196) 2024-04-19T01:59:50.643 INFO:teuthology.orchestra.run.smithi081.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-19T01:59:50.643 INFO:teuthology.orchestra.run.smithi081.stdout:19 Apr 01:59:50 ntpd[17209]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-19T01:59:50.644 INFO:teuthology.orchestra.run.smithi081.stdout:19 Apr 01:59:50 ntpd[17209]: restrict ::: KOD does nothing without LIMITED. 2024-04-19T01:59:50.644 INFO:teuthology.orchestra.run.smithi081.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-19T01:59:50.645 INFO:teuthology.orchestra.run.smithi081.stdout:19 Apr 01:59:50 ntpd[17209]: Listen and drop on 0 v6wildcard [::]:123 2024-04-19T01:59:50.645 INFO:teuthology.orchestra.run.smithi081.stdout:19 Apr 01:59:50 ntpd[17209]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-19T01:59:50.645 INFO:teuthology.orchestra.run.smithi081.stdout:19 Apr 01:59:50 ntpd[17209]: Listen normally on 2 lo 127.0.0.1:123 2024-04-19T01:59:50.645 INFO:teuthology.orchestra.run.smithi081.stdout:19 Apr 01:59:50 ntpd[17209]: Listen normally on 3 enp3s0f1 172.21.15.81:123 2024-04-19T01:59:50.645 INFO:teuthology.orchestra.run.smithi081.stdout:19 Apr 01:59:50 ntpd[17209]: Listen normally on 4 lo [::1]:123 2024-04-19T01:59:50.645 INFO:teuthology.orchestra.run.smithi081.stdout:19 Apr 01:59:50 ntpd[17209]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d753%5]:123 2024-04-19T01:59:50.645 INFO:teuthology.orchestra.run.smithi081.stdout:19 Apr 01:59:50 ntpd[17209]: Listening on routing socket on fd #22 for interface updates 2024-04-19T01:59:50.663 INFO:teuthology.orchestra.run.smithi179.stdout:19 Apr 01:59:50 ntpd[17220]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-19T01:59:50.663 INFO:teuthology.orchestra.run.smithi179.stdout:19 Apr 01:59:50 ntpd[17220]: Command line: ntpd -gq 2024-04-19T01:59:50.663 INFO:teuthology.orchestra.run.smithi179.stdout:19 Apr 01:59:50 ntpd[17220]: ---------------------------------------------------- 2024-04-19T01:59:50.663 INFO:teuthology.orchestra.run.smithi179.stdout:19 Apr 01:59:50 ntpd[17220]: ntp-4 is maintained by Network Time Foundation, 2024-04-19T01:59:50.663 INFO:teuthology.orchestra.run.smithi179.stdout:19 Apr 01:59:50 ntpd[17220]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-19T01:59:50.663 INFO:teuthology.orchestra.run.smithi179.stdout:19 Apr 01:59:50 ntpd[17220]: corporation. Support and training for ntp-4 are 2024-04-19T01:59:50.664 INFO:teuthology.orchestra.run.smithi179.stdout:19 Apr 01:59:50 ntpd[17220]: available at https://www.nwtime.org/support 2024-04-19T01:59:50.664 INFO:teuthology.orchestra.run.smithi179.stdout:19 Apr 01:59:50 ntpd[17220]: ---------------------------------------------------- 2024-04-19T01:59:50.664 INFO:teuthology.orchestra.run.smithi179.stdout:19 Apr 01:59:50 ntpd[17220]: proto: precision = 0.058 usec (-24) 2024-04-19T01:59:50.664 INFO:teuthology.orchestra.run.smithi179.stdout:19 Apr 01:59:50 ntpd[17220]: basedate set to 2022-02-04 2024-04-19T01:59:50.664 INFO:teuthology.orchestra.run.smithi179.stdout:19 Apr 01:59:50 ntpd[17220]: gps base set to 2022-02-06 (week 2196) 2024-04-19T01:59:50.665 INFO:teuthology.orchestra.run.smithi179.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-19T01:59:50.665 INFO:teuthology.orchestra.run.smithi179.stdout:19 Apr 01:59:50 ntpd[17220]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-19T01:59:50.665 INFO:teuthology.orchestra.run.smithi179.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-19T01:59:50.666 INFO:teuthology.orchestra.run.smithi179.stdout:19 Apr 01:59:50 ntpd[17220]: restrict ::: KOD does nothing without LIMITED. 2024-04-19T01:59:50.666 INFO:teuthology.orchestra.run.smithi179.stdout:19 Apr 01:59:50 ntpd[17220]: Listen and drop on 0 v6wildcard [::]:123 2024-04-19T01:59:50.666 INFO:teuthology.orchestra.run.smithi179.stdout:19 Apr 01:59:50 ntpd[17220]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-19T01:59:50.667 INFO:teuthology.orchestra.run.smithi179.stdout:19 Apr 01:59:50 ntpd[17220]: Listen normally on 2 lo 127.0.0.1:123 2024-04-19T01:59:50.667 INFO:teuthology.orchestra.run.smithi179.stdout:19 Apr 01:59:50 ntpd[17220]: Listen normally on 3 enp3s0f1 172.21.15.179:123 2024-04-19T01:59:50.667 INFO:teuthology.orchestra.run.smithi179.stdout:19 Apr 01:59:50 ntpd[17220]: Listen normally on 4 lo [::1]:123 2024-04-19T01:59:50.667 INFO:teuthology.orchestra.run.smithi179.stdout:19 Apr 01:59:50 ntpd[17220]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:725b%5]:123 2024-04-19T01:59:50.667 INFO:teuthology.orchestra.run.smithi179.stdout:19 Apr 01:59:50 ntpd[17220]: Listening on routing socket on fd #22 for interface updates 2024-04-19T01:59:51.639 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:51 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:51.639 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:51 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:51.639 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:51 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:51.639 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:51 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:51.639 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:51 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:51.639 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:51 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:51.639 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:51 ntpd[17082]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:59:51.639 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:51 ntpd[17082]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:59:51.639 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:51 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:51.639 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:51 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:51.639 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:51 ntpd[17082]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:59:51.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:51 ntpd[17082]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:59:51.641 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:51 ntpd[17209]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:51.641 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:51 ntpd[17209]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:51.641 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:51 ntpd[17209]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:51.641 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:51 ntpd[17209]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:51.641 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:51 ntpd[17209]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:51.641 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:51 ntpd[17209]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:51.641 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:51 ntpd[17209]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:59:51.641 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:51 ntpd[17209]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:59:51.641 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:51 ntpd[17209]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:51.642 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:51 ntpd[17209]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:51.642 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:51 ntpd[17209]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:59:51.642 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:51 ntpd[17209]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:59:51.664 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:51 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:51.664 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:51 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:51.664 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:51 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:51.664 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:51 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:51.664 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:51 ntpd[17220]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:59:51.664 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:51 ntpd[17220]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:59:53.639 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:53 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:53.639 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:53 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:53.639 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:53 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:53.639 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:53 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:53.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:53 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:53.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:53 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:53.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:53 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:53.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:53 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:53.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:53 ntpd[17082]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:59:53.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:53 ntpd[17082]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:59:53.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:53 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:53.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:53 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:53.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:53 ntpd[17082]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:59:53.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:53 ntpd[17082]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:59:53.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:53 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:53.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:53 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:53.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:53 ntpd[17082]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:59:53.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:53 ntpd[17082]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:59:53.642 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:53 ntpd[17209]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:53.642 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:53 ntpd[17209]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:53.642 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:53 ntpd[17209]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:53.642 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:53 ntpd[17209]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:53.642 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:53 ntpd[17209]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:53.642 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:53 ntpd[17209]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:53.642 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:53 ntpd[17209]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:59:53.642 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:53 ntpd[17209]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:59:53.642 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:53 ntpd[17209]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:53.643 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:53 ntpd[17209]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:53.643 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:53 ntpd[17209]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:59:53.643 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:53 ntpd[17209]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:59:53.665 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:53 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:53.665 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:53 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:53.665 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:53 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:53.665 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:53 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:53.665 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:53 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:53.665 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:53 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:53.666 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:53 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:53.666 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:53 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:53.666 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:53 ntpd[17220]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:59:53.666 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:53 ntpd[17220]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:59:53.666 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:53 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:53.666 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:53 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:53.666 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:53 ntpd[17220]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:59:53.666 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:53 ntpd[17220]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:59:53.666 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:53 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:53.666 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:53 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:53.666 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:53 ntpd[17220]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:59:53.666 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:53 ntpd[17220]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:59:54.638 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:54 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:54.638 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:54 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:54.640 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:54 ntpd[17209]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:54.640 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:54 ntpd[17209]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:54.640 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:54 ntpd[17209]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:54.640 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:54 ntpd[17209]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:54.640 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:54 ntpd[17209]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:54.640 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:54 ntpd[17209]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:54.640 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:54 ntpd[17209]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:59:54.640 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:54 ntpd[17209]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:59:54.664 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:54 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:54.664 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:54 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:55.639 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:55 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:55.639 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:55 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:55.639 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:55 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:55.639 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:55 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:55.639 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:55 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:55.639 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:55 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:55.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:55 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:55.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:55 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:55.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:55 ntpd[17082]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:59:55.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:55 ntpd[17082]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:59:55.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:55 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:55.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:55 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:55.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:55 ntpd[17082]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:59:55.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:55 ntpd[17082]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:59:55.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:55 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:55.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:55 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:55.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:55 ntpd[17082]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:59:55.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:55 ntpd[17082]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:59:55.642 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:55 ntpd[17209]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:55.642 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:55 ntpd[17209]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:55.642 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:55 ntpd[17209]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:55.642 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:55 ntpd[17209]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:55.642 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:55 ntpd[17209]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:55.642 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:55 ntpd[17209]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:55.642 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:55 ntpd[17209]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:59:55.643 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:55 ntpd[17209]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:59:55.643 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:55 ntpd[17209]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:55.643 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:55 ntpd[17209]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:55.643 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:55 ntpd[17209]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:59:55.643 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:55 ntpd[17209]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:59:55.665 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:55 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:55.665 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:55 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:55.665 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:55 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:55.665 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:55 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:55.665 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:55 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:55.665 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:55 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:55.666 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:55 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:55.666 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:55 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:55.666 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:55 ntpd[17220]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:59:55.666 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:55 ntpd[17220]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:59:55.666 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:55 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:55.666 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:55 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:55.666 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:55 ntpd[17220]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:59:55.666 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:55 ntpd[17220]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:59:55.666 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:55 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:55.666 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:55 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:55.666 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:55 ntpd[17220]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:59:55.666 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:55 ntpd[17220]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:59:56.639 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:56 ntpd[17209]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:56.640 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:56 ntpd[17209]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:56.640 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:56 ntpd[17209]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:56.640 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:56 ntpd[17209]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:56.640 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:56 ntpd[17209]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:59:56.640 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:56 ntpd[17209]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:59:57.639 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:57 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:57.639 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:57 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:57.639 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:57 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:57.639 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:57 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:57.639 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:57 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:57.639 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:57 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:57.639 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:57 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:57.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:57 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:57.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:57 ntpd[17082]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:59:57.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:57 ntpd[17082]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:59:57.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:57 ntpd[17082]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:57.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:57 ntpd[17082]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:57.640 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:57 ntpd[17082]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:59:57.640 INFO:teuthology.orchestra.run.smithi059.stdout:19 Apr 01:59:57 ntpd[17082]: ntpd: time slew -0.000089 s 2024-04-19T01:59:57.640 INFO:teuthology.orchestra.run.smithi059.stdout:ntpd: time slew -0.000089s 2024-04-19T01:59:57.641 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:57 ntpd[17082]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:59:57.641 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:57 ntpd[17082]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-19T01:59:57.641 INFO:teuthology.orchestra.run.smithi059.stderr:19 Apr 01:59:57 ntpd[17082]: can't open /var/log/ntpstats/loopstats.20240419: Permission denied 2024-04-19T01:59:57.642 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:57 ntpd[17209]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:57.642 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:57 ntpd[17209]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:57.642 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:57 ntpd[17209]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:57.642 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:57 ntpd[17209]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:57.642 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:57 ntpd[17209]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:57.642 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:57 ntpd[17209]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:57.643 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:57 ntpd[17209]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:59:57.643 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:57 ntpd[17209]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:59:57.643 INFO:teuthology.orchestra.run.smithi081.stdout:19 Apr 01:59:57 ntpd[17209]: ntpd: time slew -0.002746 s 2024-04-19T01:59:57.643 INFO:teuthology.orchestra.run.smithi081.stdout:ntpd: time slew -0.002746s 2024-04-19T01:59:57.643 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:57 ntpd[17209]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-19T01:59:57.643 INFO:teuthology.orchestra.run.smithi081.stderr:19 Apr 01:59:57 ntpd[17209]: can't open /var/log/ntpstats/loopstats.20240419: Permission denied 2024-04-19T01:59:57.664 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:57 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:57.665 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:57 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:57.665 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:57 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:57.665 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:57 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:57.665 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:57 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:57.665 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:57 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:57.665 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:57 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:59:57.665 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:57 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:59:57.665 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:57 ntpd[17220]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:59:57.665 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:57 ntpd[17220]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:59:57.665 INFO:teuthology.orchestra.run.smithi179.stdout:19 Apr 01:59:57 ntpd[17220]: ntpd: time slew -0.001708 s 2024-04-19T01:59:57.665 INFO:teuthology.orchestra.run.smithi179.stdout:ntpd: time slew -0.001708s 2024-04-19T01:59:57.666 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:57 ntpd[17220]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-19T01:59:57.666 INFO:teuthology.orchestra.run.smithi179.stderr:19 Apr 01:59:57 ntpd[17220]: can't open /var/log/ntpstats/loopstats.20240419: Permission denied 2024-04-19T01:59:57.700 INFO:teuthology.orchestra.run.smithi059.stdout: remote refid st t when poll reach delay offset jitter 2024-04-19T01:59:57.700 INFO:teuthology.orchestra.run.smithi059.stdout:============================================================================== 2024-04-19T01:59:57.700 INFO:teuthology.orchestra.run.smithi059.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:59:57.700 INFO:teuthology.orchestra.run.smithi059.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:59:57.700 INFO:teuthology.orchestra.run.smithi059.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:59:57.700 INFO:teuthology.orchestra.run.smithi059.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:59:57.702 INFO:teuthology.orchestra.run.smithi081.stdout: remote refid st t when poll reach delay offset jitter 2024-04-19T01:59:57.702 INFO:teuthology.orchestra.run.smithi081.stdout:============================================================================== 2024-04-19T01:59:57.702 INFO:teuthology.orchestra.run.smithi081.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:59:57.702 INFO:teuthology.orchestra.run.smithi081.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:59:57.702 INFO:teuthology.orchestra.run.smithi081.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:59:57.702 INFO:teuthology.orchestra.run.smithi081.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:59:57.718 INFO:teuthology.orchestra.run.smithi179.stdout: remote refid st t when poll reach delay offset jitter 2024-04-19T01:59:57.718 INFO:teuthology.orchestra.run.smithi179.stdout:============================================================================== 2024-04-19T01:59:57.718 INFO:teuthology.orchestra.run.smithi179.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:59:57.719 INFO:teuthology.orchestra.run.smithi179.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:59:57.719 INFO:teuthology.orchestra.run.smithi179.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:59:57.719 INFO:teuthology.orchestra.run.smithi179.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:59:57.720 INFO:teuthology.run_tasks:Running task install... 2024-04-19T01:59:57.733 DEBUG:teuthology.task.install:project ceph 2024-04-19T01:59:57.733 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1a6a3e742fd81c6d70be70552499bff767652c32'}} 2024-04-19T01:59:57.733 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '1a6a3e742fd81c6d70be70552499bff767652c32'} 2024-04-19T01:59:57.733 INFO:teuthology.task.install:Using flavor: default 2024-04-19T01:59:57.739 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-19T01:59:57.740 INFO:teuthology.task.install:extra packages: [] 2024-04-19T01:59:57.740 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt-key list | grep Ceph 2024-04-19T01:59:57.741 DEBUG:teuthology.orchestra.run.smithi081:> sudo apt-key list | grep Ceph 2024-04-19T01:59:57.741 DEBUG:teuthology.orchestra.run.smithi179:> sudo apt-key list | grep Ceph 2024-04-19T01:59:57.804 INFO:teuthology.orchestra.run.smithi059.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-19T01:59:57.804 INFO:teuthology.orchestra.run.smithi081.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-19T01:59:57.814 INFO:teuthology.orchestra.run.smithi179.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-19T01:59:57.832 INFO:teuthology.orchestra.run.smithi081.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-19T01:59:57.832 INFO:teuthology.orchestra.run.smithi081.stdout:uid [ unknown] Ceph.com (release key) 2024-04-19T01:59:57.833 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-19T01:59:57.833 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1a6a3e742fd81c6d70be70552499bff767652c32 2024-04-19T01:59:57.839 INFO:teuthology.orchestra.run.smithi059.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-19T01:59:57.839 INFO:teuthology.orchestra.run.smithi059.stdout:uid [ unknown] Ceph.com (release key) 2024-04-19T01:59:57.839 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-19T01:59:57.839 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1a6a3e742fd81c6d70be70552499bff767652c32 2024-04-19T01:59:57.843 INFO:teuthology.orchestra.run.smithi179.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-19T01:59:57.843 INFO:teuthology.orchestra.run.smithi179.stdout:uid [ unknown] Ceph.com (release key) 2024-04-19T01:59:57.843 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-19T01:59:57.843 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1a6a3e742fd81c6d70be70552499bff767652c32 2024-04-19T01:59:57.990 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default/ 2024-04-19T01:59:58.393 INFO:teuthology.task.install.deb:Package version is 19.0.0-2369-g1a6a3e74-1jammy 2024-04-19T01:59:58.398 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default/ 2024-04-19T01:59:58.398 INFO:teuthology.task.install.deb:Package version is 19.0.0-2369-g1a6a3e74-1jammy 2024-04-19T01:59:58.454 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default/ 2024-04-19T01:59:58.454 INFO:teuthology.task.install.deb:Package version is 19.0.0-2369-g1a6a3e74-1jammy 2024-04-19T01:59:58.664 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-19T01:59:58.665 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-19T01:59:58.679 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt-get update 2024-04-19T01:59:58.724 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-19T01:59:58.724 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-19T01:59:58.728 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-04-19T01:59:58.728 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-19T01:59:58.742 DEBUG:teuthology.orchestra.run.smithi179:> sudo apt-get update 2024-04-19T01:59:58.774 DEBUG:teuthology.orchestra.run.smithi081:> sudo apt-get update 2024-04-19T01:59:58.912 INFO:teuthology.orchestra.run.smithi179.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T01:59:58.912 INFO:teuthology.orchestra.run.smithi179.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-19T01:59:58.924 INFO:teuthology.orchestra.run.smithi179.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-19T01:59:58.945 INFO:teuthology.orchestra.run.smithi179.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-19T01:59:58.949 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-19T01:59:58.966 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T01:59:58.977 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T01:59:58.977 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-19T01:59:58.986 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-19T01:59:59.007 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-19T01:59:59.049 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-19T01:59:59.071 INFO:teuthology.orchestra.run.smithi059.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy InRelease 2024-04-19T01:59:59.131 INFO:teuthology.orchestra.run.smithi179.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy InRelease 2024-04-19T01:59:59.131 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-19T01:59:59.137 INFO:teuthology.orchestra.run.smithi059.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-19T01:59:59.172 INFO:teuthology.orchestra.run.smithi081.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy InRelease 2024-04-19T01:59:59.180 INFO:teuthology.orchestra.run.smithi179.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-19T01:59:59.180 INFO:teuthology.orchestra.run.smithi059.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-19T01:59:59.210 INFO:teuthology.orchestra.run.smithi081.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-19T01:59:59.218 INFO:teuthology.orchestra.run.smithi059.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,425 B] 2024-04-19T01:59:59.218 INFO:teuthology.orchestra.run.smithi179.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-19T01:59:59.232 INFO:teuthology.orchestra.run.smithi081.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-19T01:59:59.239 INFO:teuthology.orchestra.run.smithi179.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,425 B] 2024-04-19T01:59:59.245 INFO:teuthology.orchestra.run.smithi059.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-19T01:59:59.253 INFO:teuthology.orchestra.run.smithi081.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,425 B] 2024-04-19T01:59:59.266 INFO:teuthology.orchestra.run.smithi179.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-19T01:59:59.280 INFO:teuthology.orchestra.run.smithi081.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-19T01:59:59.366 INFO:teuthology.orchestra.run.smithi059.stdout:Fetched 29.0 kB in 1s (57.4 kB/s) 2024-04-19T01:59:59.371 INFO:teuthology.orchestra.run.smithi179.stdout:Fetched 29.0 kB in 0s (65.3 kB/s) 2024-04-19T01:59:59.379 INFO:teuthology.orchestra.run.smithi081.stdout:Fetched 29.0 kB in 0s (68.9 kB/s) 2024-04-19T02:00:00.328 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2024-04-19T02:00:00.344 DEBUG:teuthology.orchestra.run.smithi059:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2369-g1a6a3e74-1jammy cephadm=19.0.0-2369-g1a6a3e74-1jammy ceph-mds=19.0.0-2369-g1a6a3e74-1jammy ceph-mgr=19.0.0-2369-g1a6a3e74-1jammy ceph-common=19.0.0-2369-g1a6a3e74-1jammy ceph-fuse=19.0.0-2369-g1a6a3e74-1jammy ceph-test=19.0.0-2369-g1a6a3e74-1jammy ceph-volume=19.0.0-2369-g1a6a3e74-1jammy radosgw=19.0.0-2369-g1a6a3e74-1jammy python3-rados=19.0.0-2369-g1a6a3e74-1jammy python3-rgw=19.0.0-2369-g1a6a3e74-1jammy python3-cephfs=19.0.0-2369-g1a6a3e74-1jammy python3-rbd=19.0.0-2369-g1a6a3e74-1jammy libcephfs2=19.0.0-2369-g1a6a3e74-1jammy libcephfs-dev=19.0.0-2369-g1a6a3e74-1jammy librados2=19.0.0-2369-g1a6a3e74-1jammy librbd1=19.0.0-2369-g1a6a3e74-1jammy rbd-fuse=19.0.0-2369-g1a6a3e74-1jammy 2024-04-19T02:00:00.353 INFO:teuthology.orchestra.run.smithi081.stdout:Reading package lists... 2024-04-19T02:00:00.369 DEBUG:teuthology.orchestra.run.smithi081:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2369-g1a6a3e74-1jammy cephadm=19.0.0-2369-g1a6a3e74-1jammy ceph-mds=19.0.0-2369-g1a6a3e74-1jammy ceph-mgr=19.0.0-2369-g1a6a3e74-1jammy ceph-common=19.0.0-2369-g1a6a3e74-1jammy ceph-fuse=19.0.0-2369-g1a6a3e74-1jammy ceph-test=19.0.0-2369-g1a6a3e74-1jammy ceph-volume=19.0.0-2369-g1a6a3e74-1jammy radosgw=19.0.0-2369-g1a6a3e74-1jammy python3-rados=19.0.0-2369-g1a6a3e74-1jammy python3-rgw=19.0.0-2369-g1a6a3e74-1jammy python3-cephfs=19.0.0-2369-g1a6a3e74-1jammy python3-rbd=19.0.0-2369-g1a6a3e74-1jammy libcephfs2=19.0.0-2369-g1a6a3e74-1jammy libcephfs-dev=19.0.0-2369-g1a6a3e74-1jammy librados2=19.0.0-2369-g1a6a3e74-1jammy librbd1=19.0.0-2369-g1a6a3e74-1jammy rbd-fuse=19.0.0-2369-g1a6a3e74-1jammy 2024-04-19T02:00:00.370 INFO:teuthology.orchestra.run.smithi179.stdout:Reading package lists... 2024-04-19T02:00:00.386 DEBUG:teuthology.orchestra.run.smithi179:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2369-g1a6a3e74-1jammy cephadm=19.0.0-2369-g1a6a3e74-1jammy ceph-mds=19.0.0-2369-g1a6a3e74-1jammy ceph-mgr=19.0.0-2369-g1a6a3e74-1jammy ceph-common=19.0.0-2369-g1a6a3e74-1jammy ceph-fuse=19.0.0-2369-g1a6a3e74-1jammy ceph-test=19.0.0-2369-g1a6a3e74-1jammy ceph-volume=19.0.0-2369-g1a6a3e74-1jammy radosgw=19.0.0-2369-g1a6a3e74-1jammy python3-rados=19.0.0-2369-g1a6a3e74-1jammy python3-rgw=19.0.0-2369-g1a6a3e74-1jammy python3-cephfs=19.0.0-2369-g1a6a3e74-1jammy python3-rbd=19.0.0-2369-g1a6a3e74-1jammy libcephfs2=19.0.0-2369-g1a6a3e74-1jammy libcephfs-dev=19.0.0-2369-g1a6a3e74-1jammy librados2=19.0.0-2369-g1a6a3e74-1jammy librbd1=19.0.0-2369-g1a6a3e74-1jammy rbd-fuse=19.0.0-2369-g1a6a3e74-1jammy 2024-04-19T02:00:00.398 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2024-04-19T02:00:00.429 INFO:teuthology.orchestra.run.smithi081.stdout:Reading package lists... 2024-04-19T02:00:00.437 INFO:teuthology.orchestra.run.smithi179.stdout:Reading package lists... 2024-04-19T02:00:00.580 INFO:teuthology.orchestra.run.smithi059.stdout:Building dependency tree... 2024-04-19T02:00:00.581 INFO:teuthology.orchestra.run.smithi059.stdout:Reading state information... 2024-04-19T02:00:00.615 INFO:teuthology.orchestra.run.smithi179.stdout:Building dependency tree... 2024-04-19T02:00:00.615 INFO:teuthology.orchestra.run.smithi179.stdout:Reading state information... 2024-04-19T02:00:00.630 INFO:teuthology.orchestra.run.smithi081.stdout:Building dependency tree... 2024-04-19T02:00:00.631 INFO:teuthology.orchestra.run.smithi081.stdout:Reading state information... 2024-04-19T02:00:00.764 INFO:teuthology.orchestra.run.smithi059.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T02:00:00.764 INFO:teuthology.orchestra.run.smithi059.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T02:00:00.764 INFO:teuthology.orchestra.run.smithi059.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T02:00:00.764 INFO:teuthology.orchestra.run.smithi059.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T02:00:00.764 INFO:teuthology.orchestra.run.smithi059.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-19T02:00:00.764 INFO:teuthology.orchestra.run.smithi059.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-19T02:00:00.765 INFO:teuthology.orchestra.run.smithi059.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T02:00:00.765 INFO:teuthology.orchestra.run.smithi059.stdout:The following additional packages will be installed: 2024-04-19T02:00:00.765 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-19T02:00:00.765 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-19T02:00:00.765 INFO:teuthology.orchestra.run.smithi059.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-19T02:00:00.766 INFO:teuthology.orchestra.run.smithi059.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-19T02:00:00.767 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-19T02:00:00.767 INFO:teuthology.orchestra.run.smithi059.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-19T02:00:00.767 INFO:teuthology.orchestra.run.smithi059.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-19T02:00:00.767 INFO:teuthology.orchestra.run.smithi059.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-19T02:00:00.767 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-19T02:00:00.767 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-19T02:00:00.767 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-19T02:00:00.767 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-19T02:00:00.767 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-19T02:00:00.767 INFO:teuthology.orchestra.run.smithi059.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-19T02:00:00.767 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-19T02:00:00.768 INFO:teuthology.orchestra.run.smithi059.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-19T02:00:00.768 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-19T02:00:00.768 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-19T02:00:00.768 INFO:teuthology.orchestra.run.smithi059.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-19T02:00:00.768 INFO:teuthology.orchestra.run.smithi059.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-19T02:00:00.768 INFO:teuthology.orchestra.run.smithi059.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-19T02:00:00.768 INFO:teuthology.orchestra.run.smithi059.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-19T02:00:00.768 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-19T02:00:00.768 INFO:teuthology.orchestra.run.smithi059.stdout: xmlstarlet zip 2024-04-19T02:00:00.768 INFO:teuthology.orchestra.run.smithi059.stdout:Suggested packages: 2024-04-19T02:00:00.768 INFO:teuthology.orchestra.run.smithi059.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-19T02:00:00.769 INFO:teuthology.orchestra.run.smithi059.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-19T02:00:00.769 INFO:teuthology.orchestra.run.smithi059.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-19T02:00:00.769 INFO:teuthology.orchestra.run.smithi059.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-19T02:00:00.769 INFO:teuthology.orchestra.run.smithi059.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-19T02:00:00.769 INFO:teuthology.orchestra.run.smithi059.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-19T02:00:00.769 INFO:teuthology.orchestra.run.smithi059.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-19T02:00:00.769 INFO:teuthology.orchestra.run.smithi059.stdout:Recommended packages: 2024-04-19T02:00:00.769 INFO:teuthology.orchestra.run.smithi059.stdout: btrfs-tools 2024-04-19T02:00:00.788 INFO:teuthology.orchestra.run.smithi179.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T02:00:00.788 INFO:teuthology.orchestra.run.smithi179.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T02:00:00.788 INFO:teuthology.orchestra.run.smithi179.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T02:00:00.789 INFO:teuthology.orchestra.run.smithi179.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T02:00:00.789 INFO:teuthology.orchestra.run.smithi179.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-19T02:00:00.789 INFO:teuthology.orchestra.run.smithi179.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-19T02:00:00.789 INFO:teuthology.orchestra.run.smithi179.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T02:00:00.791 INFO:teuthology.orchestra.run.smithi179.stdout:The following additional packages will be installed: 2024-04-19T02:00:00.791 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-19T02:00:00.791 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-19T02:00:00.791 INFO:teuthology.orchestra.run.smithi179.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-19T02:00:00.791 INFO:teuthology.orchestra.run.smithi179.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-19T02:00:00.791 INFO:teuthology.orchestra.run.smithi179.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-19T02:00:00.791 INFO:teuthology.orchestra.run.smithi179.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-19T02:00:00.791 INFO:teuthology.orchestra.run.smithi179.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-19T02:00:00.791 INFO:teuthology.orchestra.run.smithi179.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-19T02:00:00.791 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-19T02:00:00.791 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-19T02:00:00.792 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-19T02:00:00.792 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-19T02:00:00.792 INFO:teuthology.orchestra.run.smithi179.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-19T02:00:00.792 INFO:teuthology.orchestra.run.smithi179.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-19T02:00:00.792 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-19T02:00:00.792 INFO:teuthology.orchestra.run.smithi179.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-19T02:00:00.792 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-19T02:00:00.792 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-19T02:00:00.792 INFO:teuthology.orchestra.run.smithi179.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-19T02:00:00.792 INFO:teuthology.orchestra.run.smithi179.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-19T02:00:00.792 INFO:teuthology.orchestra.run.smithi179.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-19T02:00:00.792 INFO:teuthology.orchestra.run.smithi179.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-19T02:00:00.792 INFO:teuthology.orchestra.run.smithi179.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-19T02:00:00.792 INFO:teuthology.orchestra.run.smithi179.stdout: xmlstarlet zip 2024-04-19T02:00:00.793 INFO:teuthology.orchestra.run.smithi179.stdout:Suggested packages: 2024-04-19T02:00:00.793 INFO:teuthology.orchestra.run.smithi179.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-19T02:00:00.793 INFO:teuthology.orchestra.run.smithi179.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-19T02:00:00.793 INFO:teuthology.orchestra.run.smithi179.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-19T02:00:00.793 INFO:teuthology.orchestra.run.smithi179.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-19T02:00:00.793 INFO:teuthology.orchestra.run.smithi179.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-19T02:00:00.793 INFO:teuthology.orchestra.run.smithi179.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-19T02:00:00.793 INFO:teuthology.orchestra.run.smithi179.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-19T02:00:00.793 INFO:teuthology.orchestra.run.smithi179.stdout:Recommended packages: 2024-04-19T02:00:00.794 INFO:teuthology.orchestra.run.smithi179.stdout: btrfs-tools 2024-04-19T02:00:00.826 INFO:teuthology.orchestra.run.smithi081.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T02:00:00.827 INFO:teuthology.orchestra.run.smithi081.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T02:00:00.827 INFO:teuthology.orchestra.run.smithi081.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T02:00:00.827 INFO:teuthology.orchestra.run.smithi081.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T02:00:00.827 INFO:teuthology.orchestra.run.smithi081.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-19T02:00:00.827 INFO:teuthology.orchestra.run.smithi081.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-19T02:00:00.827 INFO:teuthology.orchestra.run.smithi081.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T02:00:00.829 INFO:teuthology.orchestra.run.smithi081.stdout:The following additional packages will be installed: 2024-04-19T02:00:00.829 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-19T02:00:00.830 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-19T02:00:00.830 INFO:teuthology.orchestra.run.smithi081.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-19T02:00:00.830 INFO:teuthology.orchestra.run.smithi081.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-19T02:00:00.830 INFO:teuthology.orchestra.run.smithi081.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-19T02:00:00.830 INFO:teuthology.orchestra.run.smithi081.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-19T02:00:00.830 INFO:teuthology.orchestra.run.smithi081.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-19T02:00:00.830 INFO:teuthology.orchestra.run.smithi081.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-19T02:00:00.830 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-19T02:00:00.830 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-19T02:00:00.831 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-19T02:00:00.831 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-19T02:00:00.831 INFO:teuthology.orchestra.run.smithi081.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-19T02:00:00.831 INFO:teuthology.orchestra.run.smithi081.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-19T02:00:00.831 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-19T02:00:00.831 INFO:teuthology.orchestra.run.smithi081.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-19T02:00:00.831 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-19T02:00:00.831 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-19T02:00:00.831 INFO:teuthology.orchestra.run.smithi081.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-19T02:00:00.831 INFO:teuthology.orchestra.run.smithi081.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-19T02:00:00.831 INFO:teuthology.orchestra.run.smithi081.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-19T02:00:00.831 INFO:teuthology.orchestra.run.smithi081.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-19T02:00:00.832 INFO:teuthology.orchestra.run.smithi081.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-19T02:00:00.832 INFO:teuthology.orchestra.run.smithi081.stdout: xmlstarlet zip 2024-04-19T02:00:00.832 INFO:teuthology.orchestra.run.smithi081.stdout:Suggested packages: 2024-04-19T02:00:00.832 INFO:teuthology.orchestra.run.smithi081.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-19T02:00:00.832 INFO:teuthology.orchestra.run.smithi081.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-19T02:00:00.832 INFO:teuthology.orchestra.run.smithi081.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-19T02:00:00.832 INFO:teuthology.orchestra.run.smithi081.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-19T02:00:00.832 INFO:teuthology.orchestra.run.smithi081.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-19T02:00:00.832 INFO:teuthology.orchestra.run.smithi081.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-19T02:00:00.832 INFO:teuthology.orchestra.run.smithi081.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-19T02:00:00.832 INFO:teuthology.orchestra.run.smithi081.stdout:Recommended packages: 2024-04-19T02:00:00.832 INFO:teuthology.orchestra.run.smithi081.stdout: btrfs-tools 2024-04-19T02:00:00.873 INFO:teuthology.orchestra.run.smithi059.stdout:The following NEW packages will be installed: 2024-04-19T02:00:00.873 INFO:teuthology.orchestra.run.smithi059.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-19T02:00:00.873 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-19T02:00:00.873 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-19T02:00:00.873 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-19T02:00:00.873 INFO:teuthology.orchestra.run.smithi059.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-19T02:00:00.873 INFO:teuthology.orchestra.run.smithi059.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-19T02:00:00.874 INFO:teuthology.orchestra.run.smithi059.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-19T02:00:00.874 INFO:teuthology.orchestra.run.smithi059.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-19T02:00:00.875 INFO:teuthology.orchestra.run.smithi059.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-19T02:00:00.875 INFO:teuthology.orchestra.run.smithi059.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-19T02:00:00.875 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-19T02:00:00.875 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-19T02:00:00.875 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-19T02:00:00.875 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-19T02:00:00.875 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-19T02:00:00.875 INFO:teuthology.orchestra.run.smithi059.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-19T02:00:00.875 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-19T02:00:00.875 INFO:teuthology.orchestra.run.smithi059.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-19T02:00:00.875 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-19T02:00:00.875 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-19T02:00:00.875 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-19T02:00:00.875 INFO:teuthology.orchestra.run.smithi059.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-19T02:00:00.875 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-19T02:00:00.876 INFO:teuthology.orchestra.run.smithi059.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-19T02:00:00.876 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-19T02:00:00.876 INFO:teuthology.orchestra.run.smithi059.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-19T02:00:00.892 INFO:teuthology.orchestra.run.smithi179.stdout:The following NEW packages will be installed: 2024-04-19T02:00:00.892 INFO:teuthology.orchestra.run.smithi179.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-19T02:00:00.892 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-19T02:00:00.892 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-19T02:00:00.892 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-19T02:00:00.892 INFO:teuthology.orchestra.run.smithi179.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-19T02:00:00.892 INFO:teuthology.orchestra.run.smithi179.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-19T02:00:00.893 INFO:teuthology.orchestra.run.smithi179.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-19T02:00:00.894 INFO:teuthology.orchestra.run.smithi179.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-19T02:00:00.894 INFO:teuthology.orchestra.run.smithi179.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-19T02:00:00.894 INFO:teuthology.orchestra.run.smithi179.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-19T02:00:00.894 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-19T02:00:00.894 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-19T02:00:00.894 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-19T02:00:00.894 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-19T02:00:00.894 INFO:teuthology.orchestra.run.smithi179.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-19T02:00:00.894 INFO:teuthology.orchestra.run.smithi179.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-19T02:00:00.894 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-19T02:00:00.894 INFO:teuthology.orchestra.run.smithi179.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-19T02:00:00.894 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-19T02:00:00.895 INFO:teuthology.orchestra.run.smithi179.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-19T02:00:00.895 INFO:teuthology.orchestra.run.smithi179.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-19T02:00:00.895 INFO:teuthology.orchestra.run.smithi179.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-19T02:00:00.895 INFO:teuthology.orchestra.run.smithi179.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-19T02:00:00.895 INFO:teuthology.orchestra.run.smithi179.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-19T02:00:00.895 INFO:teuthology.orchestra.run.smithi179.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-19T02:00:00.895 INFO:teuthology.orchestra.run.smithi179.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-19T02:00:00.929 INFO:teuthology.orchestra.run.smithi081.stdout:The following NEW packages will be installed: 2024-04-19T02:00:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-19T02:00:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-19T02:00:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-19T02:00:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-19T02:00:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-19T02:00:00.929 INFO:teuthology.orchestra.run.smithi081.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-19T02:00:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-19T02:00:00.930 INFO:teuthology.orchestra.run.smithi081.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-19T02:00:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-19T02:00:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-19T02:00:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-19T02:00:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-19T02:00:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-19T02:00:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-19T02:00:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-19T02:00:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-19T02:00:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-19T02:00:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-19T02:00:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-19T02:00:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-19T02:00:00.931 INFO:teuthology.orchestra.run.smithi081.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-19T02:00:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-19T02:00:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-19T02:00:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-19T02:00:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-19T02:00:00.932 INFO:teuthology.orchestra.run.smithi081.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-19T02:00:00.958 INFO:teuthology.orchestra.run.smithi179.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-19T02:00:00.958 INFO:teuthology.orchestra.run.smithi179.stdout:Need to get 179 MB of archives. 2024-04-19T02:00:00.958 INFO:teuthology.orchestra.run.smithi179.stdout:After this operation, 821 MB of additional disk space will be used. 2024-04-19T02:00:00.958 INFO:teuthology.orchestra.run.smithi179.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-19T02:00:01.018 INFO:teuthology.orchestra.run.smithi059.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-19T02:00:01.018 INFO:teuthology.orchestra.run.smithi059.stdout:Need to get 179 MB of archives. 2024-04-19T02:00:01.019 INFO:teuthology.orchestra.run.smithi059.stdout:After this operation, 821 MB of additional disk space will be used. 2024-04-19T02:00:01.019 INFO:teuthology.orchestra.run.smithi059.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2369-g1a6a3e74-1jammy [3,547 kB] 2024-04-19T02:00:01.036 INFO:teuthology.orchestra.run.smithi179.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2369-g1a6a3e74-1jammy [3,547 kB] 2024-04-19T02:00:01.058 INFO:teuthology.orchestra.run.smithi179.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-19T02:00:01.061 INFO:teuthology.orchestra.run.smithi179.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-19T02:00:01.063 INFO:teuthology.orchestra.run.smithi059.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-19T02:00:01.081 INFO:teuthology.orchestra.run.smithi179.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-19T02:00:01.089 INFO:teuthology.orchestra.run.smithi081.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-19T02:00:01.089 INFO:teuthology.orchestra.run.smithi081.stdout:Need to get 179 MB of archives. 2024-04-19T02:00:01.089 INFO:teuthology.orchestra.run.smithi081.stdout:After this operation, 821 MB of additional disk space will be used. 2024-04-19T02:00:01.089 INFO:teuthology.orchestra.run.smithi081.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-19T02:00:01.146 INFO:teuthology.orchestra.run.smithi179.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-19T02:00:01.153 INFO:teuthology.orchestra.run.smithi179.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-19T02:00:01.164 INFO:teuthology.orchestra.run.smithi081.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2369-g1a6a3e74-1jammy [3,547 kB] 2024-04-19T02:00:01.164 INFO:teuthology.orchestra.run.smithi179.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-19T02:00:01.168 INFO:teuthology.orchestra.run.smithi179.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-19T02:00:01.169 INFO:teuthology.orchestra.run.smithi179.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-19T02:00:01.170 INFO:teuthology.orchestra.run.smithi179.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-19T02:00:01.178 INFO:teuthology.orchestra.run.smithi179.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-19T02:00:01.180 INFO:teuthology.orchestra.run.smithi179.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-19T02:00:01.182 INFO:teuthology.orchestra.run.smithi179.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-19T02:00:01.183 INFO:teuthology.orchestra.run.smithi179.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-19T02:00:01.190 INFO:teuthology.orchestra.run.smithi179.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-19T02:00:01.192 INFO:teuthology.orchestra.run.smithi179.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-19T02:00:01.194 INFO:teuthology.orchestra.run.smithi179.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-19T02:00:01.196 INFO:teuthology.orchestra.run.smithi081.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-19T02:00:01.199 INFO:teuthology.orchestra.run.smithi081.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-19T02:00:01.209 INFO:teuthology.orchestra.run.smithi179.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-19T02:00:01.210 INFO:teuthology.orchestra.run.smithi179.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-19T02:00:01.210 INFO:teuthology.orchestra.run.smithi179.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-19T02:00:01.211 INFO:teuthology.orchestra.run.smithi179.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-19T02:00:01.212 INFO:teuthology.orchestra.run.smithi179.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-19T02:00:01.212 INFO:teuthology.orchestra.run.smithi179.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-19T02:00:01.212 INFO:teuthology.orchestra.run.smithi179.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-19T02:00:01.222 INFO:teuthology.orchestra.run.smithi081.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-19T02:00:01.229 INFO:teuthology.orchestra.run.smithi179.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-19T02:00:01.229 INFO:teuthology.orchestra.run.smithi179.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-19T02:00:01.230 INFO:teuthology.orchestra.run.smithi179.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-19T02:00:01.249 INFO:teuthology.orchestra.run.smithi179.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-19T02:00:01.249 INFO:teuthology.orchestra.run.smithi179.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-19T02:00:01.250 INFO:teuthology.orchestra.run.smithi179.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-19T02:00:01.250 INFO:teuthology.orchestra.run.smithi179.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-19T02:00:01.251 INFO:teuthology.orchestra.run.smithi179.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-19T02:00:01.251 INFO:teuthology.orchestra.run.smithi179.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-19T02:00:01.252 INFO:teuthology.orchestra.run.smithi179.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-19T02:00:01.268 INFO:teuthology.orchestra.run.smithi179.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-19T02:00:01.269 INFO:teuthology.orchestra.run.smithi179.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-19T02:00:01.269 INFO:teuthology.orchestra.run.smithi179.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-19T02:00:01.290 INFO:teuthology.orchestra.run.smithi081.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-19T02:00:01.294 INFO:teuthology.orchestra.run.smithi179.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-19T02:00:01.294 INFO:teuthology.orchestra.run.smithi179.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-19T02:00:01.295 INFO:teuthology.orchestra.run.smithi179.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-19T02:00:01.295 INFO:teuthology.orchestra.run.smithi081.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-19T02:00:01.296 INFO:teuthology.orchestra.run.smithi179.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-19T02:00:01.297 INFO:teuthology.orchestra.run.smithi179.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-19T02:00:01.299 INFO:teuthology.orchestra.run.smithi179.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-19T02:00:01.300 INFO:teuthology.orchestra.run.smithi179.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-19T02:00:01.304 INFO:teuthology.orchestra.run.smithi059.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2369-g1a6a3e74-1jammy [3,168 kB] 2024-04-19T02:00:01.309 INFO:teuthology.orchestra.run.smithi081.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-19T02:00:01.314 INFO:teuthology.orchestra.run.smithi081.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-19T02:00:01.314 INFO:teuthology.orchestra.run.smithi081.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-19T02:00:01.315 INFO:teuthology.orchestra.run.smithi081.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-19T02:00:01.324 INFO:teuthology.orchestra.run.smithi081.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-19T02:00:01.327 INFO:teuthology.orchestra.run.smithi081.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-19T02:00:01.327 INFO:teuthology.orchestra.run.smithi179.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-19T02:00:01.328 INFO:teuthology.orchestra.run.smithi179.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-19T02:00:01.329 INFO:teuthology.orchestra.run.smithi081.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-19T02:00:01.330 INFO:teuthology.orchestra.run.smithi179.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-19T02:00:01.331 INFO:teuthology.orchestra.run.smithi081.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-19T02:00:01.347 INFO:teuthology.orchestra.run.smithi179.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-19T02:00:01.348 INFO:teuthology.orchestra.run.smithi179.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-19T02:00:01.350 INFO:teuthology.orchestra.run.smithi179.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-19T02:00:01.360 INFO:teuthology.orchestra.run.smithi081.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-19T02:00:01.363 INFO:teuthology.orchestra.run.smithi081.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-19T02:00:01.368 INFO:teuthology.orchestra.run.smithi081.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-19T02:00:01.371 INFO:teuthology.orchestra.run.smithi081.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-19T02:00:01.373 INFO:teuthology.orchestra.run.smithi081.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-19T02:00:01.373 INFO:teuthology.orchestra.run.smithi081.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-19T02:00:01.375 INFO:teuthology.orchestra.run.smithi081.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-19T02:00:01.375 INFO:teuthology.orchestra.run.smithi081.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-19T02:00:01.395 INFO:teuthology.orchestra.run.smithi179.stdout:Get:52 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2369-g1a6a3e74-1jammy [3,168 kB] 2024-04-19T02:00:01.402 INFO:teuthology.orchestra.run.smithi081.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-19T02:00:01.403 INFO:teuthology.orchestra.run.smithi081.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-19T02:00:01.417 INFO:teuthology.orchestra.run.smithi179.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-19T02:00:01.419 INFO:teuthology.orchestra.run.smithi179.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-19T02:00:01.421 INFO:teuthology.orchestra.run.smithi179.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-19T02:00:01.421 INFO:teuthology.orchestra.run.smithi179.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-19T02:00:01.423 INFO:teuthology.orchestra.run.smithi179.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-19T02:00:01.450 INFO:teuthology.orchestra.run.smithi059.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2369-g1a6a3e74-1jammy [711 kB] 2024-04-19T02:00:01.452 INFO:teuthology.orchestra.run.smithi081.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-19T02:00:01.453 INFO:teuthology.orchestra.run.smithi081.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-19T02:00:01.453 INFO:teuthology.orchestra.run.smithi081.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-19T02:00:01.457 INFO:teuthology.orchestra.run.smithi179.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-19T02:00:01.458 INFO:teuthology.orchestra.run.smithi081.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-19T02:00:01.459 INFO:teuthology.orchestra.run.smithi179.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-19T02:00:01.460 INFO:teuthology.orchestra.run.smithi081.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-19T02:00:01.461 INFO:teuthology.orchestra.run.smithi081.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-19T02:00:01.461 INFO:teuthology.orchestra.run.smithi179.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-19T02:00:01.462 INFO:teuthology.orchestra.run.smithi081.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-19T02:00:01.463 INFO:teuthology.orchestra.run.smithi081.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-19T02:00:01.484 INFO:teuthology.orchestra.run.smithi179.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-19T02:00:01.484 INFO:teuthology.orchestra.run.smithi179.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-19T02:00:01.485 INFO:teuthology.orchestra.run.smithi179.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-19T02:00:01.486 INFO:teuthology.orchestra.run.smithi179.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-19T02:00:01.486 INFO:teuthology.orchestra.run.smithi179.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-19T02:00:01.487 INFO:teuthology.orchestra.run.smithi179.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-19T02:00:01.489 INFO:teuthology.orchestra.run.smithi059.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-19T02:00:01.491 INFO:teuthology.orchestra.run.smithi081.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-19T02:00:01.491 INFO:teuthology.orchestra.run.smithi081.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-19T02:00:01.502 INFO:teuthology.orchestra.run.smithi179.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-19T02:00:01.503 INFO:teuthology.orchestra.run.smithi179.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-19T02:00:01.514 INFO:teuthology.orchestra.run.smithi081.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-19T02:00:01.516 INFO:teuthology.orchestra.run.smithi081.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-19T02:00:01.516 INFO:teuthology.orchestra.run.smithi081.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-19T02:00:01.527 INFO:teuthology.orchestra.run.smithi081.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-19T02:00:01.528 INFO:teuthology.orchestra.run.smithi081.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-19T02:00:01.529 INFO:teuthology.orchestra.run.smithi081.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-19T02:00:01.530 INFO:teuthology.orchestra.run.smithi081.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-19T02:00:01.533 INFO:teuthology.orchestra.run.smithi081.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-19T02:00:01.535 INFO:teuthology.orchestra.run.smithi059.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-19T02:00:01.542 INFO:teuthology.orchestra.run.smithi081.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-19T02:00:01.543 INFO:teuthology.orchestra.run.smithi081.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-19T02:00:01.549 INFO:teuthology.orchestra.run.smithi179.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-19T02:00:01.551 INFO:teuthology.orchestra.run.smithi179.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-19T02:00:01.553 INFO:teuthology.orchestra.run.smithi179.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-19T02:00:01.554 INFO:teuthology.orchestra.run.smithi179.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-19T02:00:01.557 INFO:teuthology.orchestra.run.smithi081.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-19T02:00:01.558 INFO:teuthology.orchestra.run.smithi179.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-19T02:00:01.559 INFO:teuthology.orchestra.run.smithi081.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-19T02:00:01.561 INFO:teuthology.orchestra.run.smithi179.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-19T02:00:01.563 INFO:teuthology.orchestra.run.smithi179.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-19T02:00:01.563 INFO:teuthology.orchestra.run.smithi179.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-19T02:00:01.564 INFO:teuthology.orchestra.run.smithi081.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-19T02:00:01.565 INFO:teuthology.orchestra.run.smithi081.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-19T02:00:01.567 INFO:teuthology.orchestra.run.smithi081.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-19T02:00:01.569 INFO:teuthology.orchestra.run.smithi179.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-19T02:00:01.572 INFO:teuthology.orchestra.run.smithi081.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-19T02:00:01.592 INFO:teuthology.orchestra.run.smithi179.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-19T02:00:01.593 INFO:teuthology.orchestra.run.smithi059.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2369-g1a6a3e74-1jammy [356 kB] 2024-04-19T02:00:01.601 INFO:teuthology.orchestra.run.smithi059.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2369-g1a6a3e74-1jammy [32.8 kB] 2024-04-19T02:00:01.602 INFO:teuthology.orchestra.run.smithi059.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2369-g1a6a3e74-1jammy [180 kB] 2024-04-19T02:00:01.606 INFO:teuthology.orchestra.run.smithi059.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2369-g1a6a3e74-1jammy [65.7 kB] 2024-04-19T02:00:01.608 INFO:teuthology.orchestra.run.smithi059.stdout:Get:11 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2369-g1a6a3e74-1jammy [330 kB] 2024-04-19T02:00:01.615 INFO:teuthology.orchestra.run.smithi059.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2369-g1a6a3e74-1jammy [6,775 kB] 2024-04-19T02:00:01.617 INFO:teuthology.orchestra.run.smithi179.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-19T02:00:01.617 INFO:teuthology.orchestra.run.smithi179.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-19T02:00:01.635 INFO:teuthology.orchestra.run.smithi059.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-19T02:00:01.649 INFO:teuthology.orchestra.run.smithi179.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2369-g1a6a3e74-1jammy [711 kB] 2024-04-19T02:00:01.654 INFO:teuthology.orchestra.run.smithi081.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-19T02:00:01.655 INFO:teuthology.orchestra.run.smithi081.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-19T02:00:01.657 INFO:teuthology.orchestra.run.smithi081.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-19T02:00:01.657 INFO:teuthology.orchestra.run.smithi081.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-19T02:00:01.658 INFO:teuthology.orchestra.run.smithi081.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-19T02:00:01.680 INFO:teuthology.orchestra.run.smithi179.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2369-g1a6a3e74-1jammy [356 kB] 2024-04-19T02:00:01.688 INFO:teuthology.orchestra.run.smithi081.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-19T02:00:01.690 INFO:teuthology.orchestra.run.smithi179.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-19T02:00:01.691 INFO:teuthology.orchestra.run.smithi179.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-19T02:00:01.693 INFO:teuthology.orchestra.run.smithi179.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2369-g1a6a3e74-1jammy [32.8 kB] 2024-04-19T02:00:01.694 INFO:teuthology.orchestra.run.smithi081.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-19T02:00:01.694 INFO:teuthology.orchestra.run.smithi081.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-19T02:00:01.695 INFO:teuthology.orchestra.run.smithi179.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-19T02:00:01.696 INFO:teuthology.orchestra.run.smithi179.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2369-g1a6a3e74-1jammy [180 kB] 2024-04-19T02:00:01.697 INFO:teuthology.orchestra.run.smithi179.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-19T02:00:01.703 INFO:teuthology.orchestra.run.smithi179.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2369-g1a6a3e74-1jammy [65.7 kB] 2024-04-19T02:00:01.705 INFO:teuthology.orchestra.run.smithi179.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2369-g1a6a3e74-1jammy [330 kB] 2024-04-19T02:00:01.714 INFO:teuthology.orchestra.run.smithi179.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-19T02:00:01.715 INFO:teuthology.orchestra.run.smithi179.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-19T02:00:01.715 INFO:teuthology.orchestra.run.smithi179.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-19T02:00:01.720 INFO:teuthology.orchestra.run.smithi179.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-19T02:00:01.721 INFO:teuthology.orchestra.run.smithi081.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-19T02:00:01.721 INFO:teuthology.orchestra.run.smithi081.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-19T02:00:01.722 INFO:teuthology.orchestra.run.smithi081.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-19T02:00:01.723 INFO:teuthology.orchestra.run.smithi081.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-19T02:00:01.723 INFO:teuthology.orchestra.run.smithi081.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-19T02:00:01.723 INFO:teuthology.orchestra.run.smithi081.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-19T02:00:01.729 INFO:teuthology.orchestra.run.smithi179.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-19T02:00:01.746 INFO:teuthology.orchestra.run.smithi179.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2369-g1a6a3e74-1jammy [6,775 kB] 2024-04-19T02:00:01.755 INFO:teuthology.orchestra.run.smithi081.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-19T02:00:01.756 INFO:teuthology.orchestra.run.smithi081.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-19T02:00:01.760 INFO:teuthology.orchestra.run.smithi081.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-19T02:00:01.776 INFO:teuthology.orchestra.run.smithi081.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-19T02:00:01.777 INFO:teuthology.orchestra.run.smithi081.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-19T02:00:01.778 INFO:teuthology.orchestra.run.smithi081.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-19T02:00:01.783 INFO:teuthology.orchestra.run.smithi081.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-19T02:00:01.786 INFO:teuthology.orchestra.run.smithi081.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-19T02:00:01.787 INFO:teuthology.orchestra.run.smithi081.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-19T02:00:01.787 INFO:teuthology.orchestra.run.smithi081.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-19T02:00:01.798 INFO:teuthology.orchestra.run.smithi081.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-19T02:00:01.799 INFO:teuthology.orchestra.run.smithi081.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-19T02:00:01.803 INFO:teuthology.orchestra.run.smithi081.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-19T02:00:01.819 INFO:teuthology.orchestra.run.smithi081.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-19T02:00:01.820 INFO:teuthology.orchestra.run.smithi081.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-19T02:00:01.821 INFO:teuthology.orchestra.run.smithi081.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-19T02:00:01.822 INFO:teuthology.orchestra.run.smithi081.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-19T02:00:01.823 INFO:teuthology.orchestra.run.smithi081.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-19T02:00:01.832 INFO:teuthology.orchestra.run.smithi081.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-19T02:00:01.832 INFO:teuthology.orchestra.run.smithi081.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-19T02:00:01.840 INFO:teuthology.orchestra.run.smithi081.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-19T02:00:01.843 INFO:teuthology.orchestra.run.smithi081.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-19T02:00:01.844 INFO:teuthology.orchestra.run.smithi081.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-19T02:00:01.868 INFO:teuthology.orchestra.run.smithi059.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-19T02:00:01.876 INFO:teuthology.orchestra.run.smithi059.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-19T02:00:01.906 INFO:teuthology.orchestra.run.smithi059.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-19T02:00:01.915 INFO:teuthology.orchestra.run.smithi059.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-19T02:00:01.916 INFO:teuthology.orchestra.run.smithi059.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-19T02:00:01.917 INFO:teuthology.orchestra.run.smithi059.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-19T02:00:01.936 INFO:teuthology.orchestra.run.smithi059.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-19T02:00:01.964 INFO:teuthology.orchestra.run.smithi059.stdout:Get:21 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2369-g1a6a3e74-1jammy [112 kB] 2024-04-19T02:00:01.967 INFO:teuthology.orchestra.run.smithi059.stdout:Get:22 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2369-g1a6a3e74-1jammy [466 kB] 2024-04-19T02:00:02.009 INFO:teuthology.orchestra.run.smithi059.stdout:Get:23 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2369-g1a6a3e74-1jammy [25.6 MB] 2024-04-19T02:00:02.255 INFO:teuthology.orchestra.run.smithi179.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2369-g1a6a3e74-1jammy [112 kB] 2024-04-19T02:00:02.287 INFO:teuthology.orchestra.run.smithi179.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2369-g1a6a3e74-1jammy [466 kB] 2024-04-19T02:00:02.307 INFO:teuthology.orchestra.run.smithi179.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2369-g1a6a3e74-1jammy [25.6 MB] 2024-04-19T02:00:03.650 INFO:teuthology.orchestra.run.smithi059.stdout:Get:24 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2369-g1a6a3e74-1jammy [5,084 kB] 2024-04-19T02:00:03.865 INFO:teuthology.orchestra.run.smithi059.stdout:Get:25 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2369-g1a6a3e74-1jammy [244 kB] 2024-04-19T02:00:03.870 INFO:teuthology.orchestra.run.smithi059.stdout:Get:26 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2369-g1a6a3e74-1jammy [124 kB] 2024-04-19T02:00:03.874 INFO:teuthology.orchestra.run.smithi059.stdout:Get:27 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2369-g1a6a3e74-1jammy [1,525 kB] 2024-04-19T02:00:03.980 INFO:teuthology.orchestra.run.smithi059.stdout:Get:28 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2369-g1a6a3e74-1jammy [6,246 kB] 2024-04-19T02:00:04.354 INFO:teuthology.orchestra.run.smithi059.stdout:Get:29 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2369-g1a6a3e74-1jammy [22.7 MB] 2024-04-19T02:00:04.367 INFO:teuthology.orchestra.run.smithi179.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2369-g1a6a3e74-1jammy [5,084 kB] 2024-04-19T02:00:04.681 INFO:teuthology.orchestra.run.smithi179.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2369-g1a6a3e74-1jammy [244 kB] 2024-04-19T02:00:04.722 INFO:teuthology.orchestra.run.smithi179.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2369-g1a6a3e74-1jammy [124 kB] 2024-04-19T02:00:04.726 INFO:teuthology.orchestra.run.smithi179.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2369-g1a6a3e74-1jammy [1,525 kB] 2024-04-19T02:00:04.779 INFO:teuthology.orchestra.run.smithi081.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2369-g1a6a3e74-1jammy [3,168 kB] 2024-04-19T02:00:04.828 INFO:teuthology.orchestra.run.smithi179.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2369-g1a6a3e74-1jammy [6,246 kB] 2024-04-19T02:00:05.312 INFO:teuthology.orchestra.run.smithi179.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2369-g1a6a3e74-1jammy [22.7 MB] 2024-04-19T02:00:05.802 INFO:teuthology.orchestra.run.smithi059.stdout:Get:30 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2369-g1a6a3e74-1jammy [14.0 kB] 2024-04-19T02:00:05.803 INFO:teuthology.orchestra.run.smithi059.stdout:Get:31 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2369-g1a6a3e74-1jammy [869 kB] 2024-04-19T02:00:05.820 INFO:teuthology.orchestra.run.smithi059.stdout:Get:32 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2369-g1a6a3e74-1jammy [2,435 kB] 2024-04-19T02:00:05.876 INFO:teuthology.orchestra.run.smithi059.stdout:Get:33 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2369-g1a6a3e74-1jammy [791 kB] 2024-04-19T02:00:05.927 INFO:teuthology.orchestra.run.smithi059.stdout:Get:34 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2369-g1a6a3e74-1jammy [147 kB] 2024-04-19T02:00:05.932 INFO:teuthology.orchestra.run.smithi059.stdout:Get:35 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2369-g1a6a3e74-1jammy [2,295 kB] 2024-04-19T02:00:06.060 INFO:teuthology.orchestra.run.smithi059.stdout:Get:36 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2369-g1a6a3e74-1jammy [8,625 kB] 2024-04-19T02:00:06.511 INFO:teuthology.orchestra.run.smithi059.stdout:Get:37 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2369-g1a6a3e74-1jammy [14.2 kB] 2024-04-19T02:00:06.512 INFO:teuthology.orchestra.run.smithi059.stdout:Get:38 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2369-g1a6a3e74-1jammy [50.7 MB] 2024-04-19T02:00:07.442 INFO:teuthology.orchestra.run.smithi081.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2369-g1a6a3e74-1jammy [711 kB] 2024-04-19T02:00:07.503 INFO:teuthology.orchestra.run.smithi179.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2369-g1a6a3e74-1jammy [14.0 kB] 2024-04-19T02:00:07.503 INFO:teuthology.orchestra.run.smithi179.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2369-g1a6a3e74-1jammy [869 kB] 2024-04-19T02:00:07.520 INFO:teuthology.orchestra.run.smithi179.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2369-g1a6a3e74-1jammy [2,435 kB] 2024-04-19T02:00:07.583 INFO:teuthology.orchestra.run.smithi059.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-19T02:00:07.689 INFO:teuthology.orchestra.run.smithi059.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-19T02:00:07.698 INFO:teuthology.orchestra.run.smithi059.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-19T02:00:07.698 INFO:teuthology.orchestra.run.smithi059.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-19T02:00:07.718 INFO:teuthology.orchestra.run.smithi059.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-19T02:00:07.724 INFO:teuthology.orchestra.run.smithi179.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2369-g1a6a3e74-1jammy [791 kB] 2024-04-19T02:00:07.732 INFO:teuthology.orchestra.run.smithi059.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-19T02:00:07.739 INFO:teuthology.orchestra.run.smithi059.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-19T02:00:07.741 INFO:teuthology.orchestra.run.smithi059.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-19T02:00:07.743 INFO:teuthology.orchestra.run.smithi059.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-19T02:00:07.746 INFO:teuthology.orchestra.run.smithi059.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-19T02:00:07.747 INFO:teuthology.orchestra.run.smithi059.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-19T02:00:07.748 INFO:teuthology.orchestra.run.smithi059.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-19T02:00:07.749 INFO:teuthology.orchestra.run.smithi059.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-19T02:00:07.752 INFO:teuthology.orchestra.run.smithi059.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-19T02:00:07.773 INFO:teuthology.orchestra.run.smithi059.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-19T02:00:07.773 INFO:teuthology.orchestra.run.smithi059.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-19T02:00:07.782 INFO:teuthology.orchestra.run.smithi059.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-19T02:00:07.784 INFO:teuthology.orchestra.run.smithi059.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-19T02:00:07.785 INFO:teuthology.orchestra.run.smithi059.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-19T02:00:07.787 INFO:teuthology.orchestra.run.smithi059.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-19T02:00:07.790 INFO:teuthology.orchestra.run.smithi059.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-19T02:00:07.791 INFO:teuthology.orchestra.run.smithi059.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-19T02:00:07.792 INFO:teuthology.orchestra.run.smithi059.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-19T02:00:07.796 INFO:teuthology.orchestra.run.smithi059.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-19T02:00:07.798 INFO:teuthology.orchestra.run.smithi179.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2369-g1a6a3e74-1jammy [147 kB] 2024-04-19T02:00:07.803 INFO:teuthology.orchestra.run.smithi179.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2369-g1a6a3e74-1jammy [2,295 kB] 2024-04-19T02:00:07.817 INFO:teuthology.orchestra.run.smithi059.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-19T02:00:07.818 INFO:teuthology.orchestra.run.smithi059.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-19T02:00:07.835 INFO:teuthology.orchestra.run.smithi059.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-19T02:00:07.836 INFO:teuthology.orchestra.run.smithi059.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-19T02:00:07.837 INFO:teuthology.orchestra.run.smithi059.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-19T02:00:07.838 INFO:teuthology.orchestra.run.smithi059.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-19T02:00:07.841 INFO:teuthology.orchestra.run.smithi059.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-19T02:00:07.844 INFO:teuthology.orchestra.run.smithi059.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-19T02:00:07.846 INFO:teuthology.orchestra.run.smithi059.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-19T02:00:07.848 INFO:teuthology.orchestra.run.smithi059.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-19T02:00:07.858 INFO:teuthology.orchestra.run.smithi059.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-19T02:00:07.863 INFO:teuthology.orchestra.run.smithi059.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-19T02:00:07.879 INFO:teuthology.orchestra.run.smithi059.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-19T02:00:07.881 INFO:teuthology.orchestra.run.smithi059.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-19T02:00:07.886 INFO:teuthology.orchestra.run.smithi059.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-19T02:00:07.969 INFO:teuthology.orchestra.run.smithi059.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-19T02:00:07.970 INFO:teuthology.orchestra.run.smithi059.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-19T02:00:07.972 INFO:teuthology.orchestra.run.smithi059.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-19T02:00:07.972 INFO:teuthology.orchestra.run.smithi059.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-19T02:00:07.973 INFO:teuthology.orchestra.run.smithi059.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-19T02:00:07.995 INFO:teuthology.orchestra.run.smithi059.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-19T02:00:07.997 INFO:teuthology.orchestra.run.smithi059.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-19T02:00:07.997 INFO:teuthology.orchestra.run.smithi059.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-19T02:00:08.012 INFO:teuthology.orchestra.run.smithi059.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-19T02:00:08.013 INFO:teuthology.orchestra.run.smithi059.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-19T02:00:08.014 INFO:teuthology.orchestra.run.smithi059.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-19T02:00:08.014 INFO:teuthology.orchestra.run.smithi059.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-19T02:00:08.015 INFO:teuthology.orchestra.run.smithi059.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-19T02:00:08.015 INFO:teuthology.orchestra.run.smithi059.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-19T02:00:08.017 INFO:teuthology.orchestra.run.smithi081.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2369-g1a6a3e74-1jammy [356 kB] 2024-04-19T02:00:08.025 INFO:teuthology.orchestra.run.smithi179.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2369-g1a6a3e74-1jammy [8,625 kB] 2024-04-19T02:00:08.034 INFO:teuthology.orchestra.run.smithi059.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-19T02:00:08.034 INFO:teuthology.orchestra.run.smithi059.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-19T02:00:08.038 INFO:teuthology.orchestra.run.smithi059.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-19T02:00:08.054 INFO:teuthology.orchestra.run.smithi059.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-19T02:00:08.056 INFO:teuthology.orchestra.run.smithi059.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-19T02:00:08.056 INFO:teuthology.orchestra.run.smithi059.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-19T02:00:08.060 INFO:teuthology.orchestra.run.smithi059.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-19T02:00:08.062 INFO:teuthology.orchestra.run.smithi059.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-19T02:00:08.063 INFO:teuthology.orchestra.run.smithi059.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-19T02:00:08.064 INFO:teuthology.orchestra.run.smithi059.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-19T02:00:08.075 INFO:teuthology.orchestra.run.smithi059.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-19T02:00:08.075 INFO:teuthology.orchestra.run.smithi059.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-19T02:00:08.079 INFO:teuthology.orchestra.run.smithi059.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-19T02:00:08.095 INFO:teuthology.orchestra.run.smithi059.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-19T02:00:08.096 INFO:teuthology.orchestra.run.smithi059.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-19T02:00:08.096 INFO:teuthology.orchestra.run.smithi059.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-19T02:00:08.098 INFO:teuthology.orchestra.run.smithi059.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-19T02:00:08.099 INFO:teuthology.orchestra.run.smithi059.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-19T02:00:08.106 INFO:teuthology.orchestra.run.smithi059.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-19T02:00:08.106 INFO:teuthology.orchestra.run.smithi059.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-19T02:00:08.116 INFO:teuthology.orchestra.run.smithi059.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-19T02:00:08.118 INFO:teuthology.orchestra.run.smithi059.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-19T02:00:08.119 INFO:teuthology.orchestra.run.smithi059.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-19T02:00:08.287 INFO:teuthology.orchestra.run.smithi081.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2369-g1a6a3e74-1jammy [32.8 kB] 2024-04-19T02:00:08.299 INFO:teuthology.orchestra.run.smithi081.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2369-g1a6a3e74-1jammy [180 kB] 2024-04-19T02:00:08.417 INFO:teuthology.orchestra.run.smithi081.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2369-g1a6a3e74-1jammy [65.7 kB] 2024-04-19T02:00:08.482 INFO:teuthology.orchestra.run.smithi081.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2369-g1a6a3e74-1jammy [330 kB] 2024-04-19T02:00:08.713 INFO:teuthology.orchestra.run.smithi081.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2369-g1a6a3e74-1jammy [6,775 kB] 2024-04-19T02:00:08.799 INFO:teuthology.orchestra.run.smithi179.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2369-g1a6a3e74-1jammy [14.2 kB] 2024-04-19T02:00:08.799 INFO:teuthology.orchestra.run.smithi179.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2369-g1a6a3e74-1jammy [50.7 MB] 2024-04-19T02:00:09.492 INFO:teuthology.orchestra.run.smithi059.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2369-g1a6a3e74-1jammy [121 kB] 2024-04-19T02:00:09.494 INFO:teuthology.orchestra.run.smithi059.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2369-g1a6a3e74-1jammy [38.9 kB] 2024-04-19T02:00:09.496 INFO:teuthology.orchestra.run.smithi059.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2369-g1a6a3e74-1jammy [13.4 MB] 2024-04-19T02:00:10.127 INFO:teuthology.orchestra.run.smithi059.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2369-g1a6a3e74-1jammy [91.4 kB] 2024-04-19T02:00:10.666 INFO:teuthology.orchestra.run.smithi059.stdout:Fetched 179 MB in 9s (19.4 MB/s) 2024-04-19T02:00:10.835 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-19T02:00:10.938 INFO:teuthology.orchestra.run.smithi059.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-19T02:00:10.944 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-19T02:00:11.025 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-19T02:00:11.362 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-19T02:00:11.388 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-19T02:00:11.409 INFO:teuthology.orchestra.run.smithi081.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2369-g1a6a3e74-1jammy [112 kB] 2024-04-19T02:00:11.428 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-19T02:00:11.433 INFO:teuthology.orchestra.run.smithi081.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2369-g1a6a3e74-1jammy [466 kB] 2024-04-19T02:00:11.528 INFO:teuthology.orchestra.run.smithi081.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2369-g1a6a3e74-1jammy [25.6 MB] 2024-04-19T02:00:11.672 INFO:teuthology.orchestra.run.smithi179.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2369-g1a6a3e74-1jammy [121 kB] 2024-04-19T02:00:11.675 INFO:teuthology.orchestra.run.smithi179.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2369-g1a6a3e74-1jammy [38.9 kB] 2024-04-19T02:00:11.676 INFO:teuthology.orchestra.run.smithi179.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2369-g1a6a3e74-1jammy [13.4 MB] 2024-04-19T02:00:11.789 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-19T02:00:11.816 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-19T02:00:11.856 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-19T02:00:12.057 INFO:teuthology.orchestra.run.smithi179.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2369-g1a6a3e74-1jammy [91.4 kB] 2024-04-19T02:00:12.235 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-19T02:00:12.261 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T02:00:12.301 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T02:00:12.617 INFO:teuthology.orchestra.run.smithi179.stdout:Fetched 179 MB in 11s (16.1 MB/s) 2024-04-19T02:00:12.776 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-19T02:00:12.876 INFO:teuthology.orchestra.run.smithi179.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-19T02:00:12.879 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-19T02:00:12.882 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-19T02:00:12.906 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T02:00:12.966 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-19T02:00:12.971 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T02:00:13.287 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-19T02:00:13.313 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-19T02:00:13.324 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-19T02:00:13.351 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T02:00:13.353 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-19T02:00:13.390 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T02:00:13.702 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-19T02:00:13.717 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-19T02:00:13.751 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-19T02:00:13.865 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-19T02:00:13.891 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-19T02:00:13.931 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-19T02:00:14.063 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package librados2. 2024-04-19T02:00:14.089 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../007-librados2_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:14.129 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking librados2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:14.402 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-19T02:00:14.428 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T02:00:14.476 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T02:00:14.518 INFO:teuthology.orchestra.run.smithi081.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2369-g1a6a3e74-1jammy [5,084 kB] 2024-04-19T02:00:14.591 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package librbd1. 2024-04-19T02:00:14.617 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../008-librbd1_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:14.657 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking librbd1 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:14.843 INFO:teuthology.orchestra.run.smithi081.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2369-g1a6a3e74-1jammy [244 kB] 2024-04-19T02:00:14.868 INFO:teuthology.orchestra.run.smithi081.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2369-g1a6a3e74-1jammy [124 kB] 2024-04-19T02:00:14.871 INFO:teuthology.orchestra.run.smithi081.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2369-g1a6a3e74-1jammy [1,525 kB] 2024-04-19T02:00:14.880 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-19T02:00:14.906 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T02:00:14.946 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T02:00:14.976 INFO:teuthology.orchestra.run.smithi081.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2369-g1a6a3e74-1jammy [6,246 kB] 2024-04-19T02:00:15.110 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libcephfs2. 2024-04-19T02:00:15.136 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:15.176 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libcephfs2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:15.293 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-19T02:00:15.320 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T02:00:15.366 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T02:00:15.369 INFO:teuthology.orchestra.run.smithi081.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2369-g1a6a3e74-1jammy [22.7 MB] 2024-04-19T02:00:15.462 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-rados. 2024-04-19T02:00:15.478 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:15.512 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-rados (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:15.711 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-19T02:00:15.737 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-19T02:00:15.777 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-19T02:00:15.831 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-19T02:00:15.858 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-19T02:00:15.898 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-ceph-argparse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:16.080 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package librados2. 2024-04-19T02:00:16.106 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../007-librados2_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:16.138 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking librados2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:16.184 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-cephfs. 2024-04-19T02:00:16.210 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:16.250 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-cephfs (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:16.553 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-19T02:00:16.574 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package librbd1. 2024-04-19T02:00:16.580 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-19T02:00:16.601 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../008-librbd1_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:16.619 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-ceph-common (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:16.641 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking librbd1 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:16.668 INFO:teuthology.orchestra.run.smithi081.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2369-g1a6a3e74-1jammy [14.0 kB] 2024-04-19T02:00:16.669 INFO:teuthology.orchestra.run.smithi081.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2369-g1a6a3e74-1jammy [869 kB] 2024-04-19T02:00:16.686 INFO:teuthology.orchestra.run.smithi081.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2369-g1a6a3e74-1jammy [2,435 kB] 2024-04-19T02:00:16.757 INFO:teuthology.orchestra.run.smithi081.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2369-g1a6a3e74-1jammy [791 kB] 2024-04-19T02:00:16.790 INFO:teuthology.orchestra.run.smithi081.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2369-g1a6a3e74-1jammy [147 kB] 2024-04-19T02:00:16.812 INFO:teuthology.orchestra.run.smithi081.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2369-g1a6a3e74-1jammy [2,295 kB] 2024-04-19T02:00:16.898 INFO:teuthology.orchestra.run.smithi081.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2369-g1a6a3e74-1jammy [8,625 kB] 2024-04-19T02:00:16.931 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-19T02:00:16.958 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-19T02:00:16.997 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-19T02:00:17.086 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libcephfs2. 2024-04-19T02:00:17.112 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:17.152 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libcephfs2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:17.245 INFO:teuthology.orchestra.run.smithi081.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2369-g1a6a3e74-1jammy [14.2 kB] 2024-04-19T02:00:17.246 INFO:teuthology.orchestra.run.smithi081.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2369-g1a6a3e74-1jammy [50.7 MB] 2024-04-19T02:00:17.317 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-prettytable. 2024-04-19T02:00:17.344 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-19T02:00:17.384 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-19T02:00:17.396 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-rados. 2024-04-19T02:00:17.412 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:17.446 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-rados (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:17.628 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-rbd. 2024-04-19T02:00:17.654 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:17.686 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-rbd (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:17.799 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-19T02:00:17.825 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-19T02:00:17.865 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-ceph-argparse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:18.006 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-19T02:00:18.032 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-19T02:00:18.110 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-cephfs. 2024-04-19T02:00:18.125 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:18.168 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-cephfs (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:18.225 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-19T02:00:18.479 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-19T02:00:18.506 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-19T02:00:18.546 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-ceph-common (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:18.610 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-19T02:00:18.637 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-19T02:00:18.677 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-19T02:00:18.923 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-19T02:00:18.949 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-19T02:00:18.963 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-19T02:00:18.979 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-19T02:00:19.013 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-19T02:00:19.049 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-19T02:00:19.186 INFO:teuthology.orchestra.run.smithi081.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2369-g1a6a3e74-1jammy [121 kB] 2024-04-19T02:00:19.188 INFO:teuthology.orchestra.run.smithi081.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2369-g1a6a3e74-1jammy [38.9 kB] 2024-04-19T02:00:19.189 INFO:teuthology.orchestra.run.smithi081.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2369-g1a6a3e74-1jammy [13.4 MB] 2024-04-19T02:00:19.341 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package lua5.1. 2024-04-19T02:00:19.343 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-prettytable. 2024-04-19T02:00:19.368 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-19T02:00:19.370 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-19T02:00:19.407 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-19T02:00:19.410 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-19T02:00:19.533 INFO:teuthology.orchestra.run.smithi081.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/squid/1a6a3e742fd81c6d70be70552499bff767652c32/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2369-g1a6a3e74-1jammy [91.4 kB] 2024-04-19T02:00:19.671 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-rbd. 2024-04-19T02:00:19.687 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:19.719 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package lua-any. 2024-04-19T02:00:19.729 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-rbd (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:19.746 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-19T02:00:19.786 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-19T02:00:20.013 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package zip. 2024-04-19T02:00:20.029 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-19T02:00:20.063 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking zip (3.0-12build2) ... 2024-04-19T02:00:20.082 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-19T02:00:20.098 INFO:teuthology.orchestra.run.smithi081.stdout:Fetched 179 MB in 19s (9,650 kB/s) 2024-04-19T02:00:20.109 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-19T02:00:20.149 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-19T02:00:20.259 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-19T02:00:20.359 INFO:teuthology.orchestra.run.smithi081.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-19T02:00:20.365 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-19T02:00:20.383 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package unzip. 2024-04-19T02:00:20.401 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-19T02:00:20.410 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-19T02:00:20.416 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-19T02:00:20.449 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-19T02:00:20.449 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-19T02:00:20.468 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-19T02:00:20.752 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-19T02:00:20.762 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-19T02:00:20.778 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-19T02:00:20.778 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-19T02:00:20.811 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-19T02:00:20.820 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-19T02:00:20.845 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package luarocks. 2024-04-19T02:00:20.872 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-19T02:00:20.912 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-19T02:00:21.122 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-19T02:00:21.148 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-19T02:00:21.149 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package lua5.1. 2024-04-19T02:00:21.176 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-19T02:00:21.189 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-19T02:00:21.215 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-19T02:00:21.303 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package librgw2. 2024-04-19T02:00:21.330 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../025-librgw2_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:21.386 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking librgw2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:21.527 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package lua-any. 2024-04-19T02:00:21.543 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-19T02:00:21.585 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-19T02:00:21.634 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-19T02:00:21.660 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T02:00:21.709 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T02:00:21.838 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package zip. 2024-04-19T02:00:21.865 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-19T02:00:21.904 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking zip (3.0-12build2) ... 2024-04-19T02:00:21.922 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-rgw. 2024-04-19T02:00:21.938 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:21.972 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-rgw (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:22.079 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-19T02:00:22.105 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T02:00:22.137 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T02:00:22.207 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package unzip. 2024-04-19T02:00:22.235 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-19T02:00:22.274 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-19T02:00:22.316 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-19T02:00:22.343 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-19T02:00:22.383 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-19T02:00:22.507 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-19T02:00:22.533 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T02:00:22.573 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T02:00:22.628 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package luarocks. 2024-04-19T02:00:22.655 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-19T02:00:22.686 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-19T02:00:22.719 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libradosstriper1. 2024-04-19T02:00:22.746 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:22.786 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libradosstriper1 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:22.910 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-19T02:00:22.936 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-19T02:00:22.976 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-19T02:00:23.077 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package librgw2. 2024-04-19T02:00:23.104 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../025-librgw2_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:23.144 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking librgw2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:23.271 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package librados2. 2024-04-19T02:00:23.298 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../007-librados2_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:23.338 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking librados2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:23.339 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-common. 2024-04-19T02:00:23.366 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:23.397 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-common (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:23.679 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-rgw. 2024-04-19T02:00:23.695 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:23.737 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-rgw (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:23.740 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package librbd1. 2024-04-19T02:00:23.767 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../008-librbd1_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:23.807 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking librbd1 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:24.082 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-19T02:00:24.109 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-19T02:00:24.150 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-19T02:00:24.243 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libcephfs2. 2024-04-19T02:00:24.270 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:24.310 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libcephfs2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:24.427 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libradosstriper1. 2024-04-19T02:00:24.453 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:24.485 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libradosstriper1 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:24.587 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-rados. 2024-04-19T02:00:24.602 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:24.645 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-rados (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:24.861 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-base. 2024-04-19T02:00:24.888 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:24.922 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ceph-common. 2024-04-19T02:00:24.937 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-base (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:24.949 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:24.988 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ceph-common (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:25.124 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-19T02:00:25.147 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-19T02:00:25.182 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-ceph-argparse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:25.418 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-cephfs. 2024-04-19T02:00:25.444 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:25.484 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-cephfs (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:25.507 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-19T02:00:25.534 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-19T02:00:25.573 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-19T02:00:25.795 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-19T02:00:25.822 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-19T02:00:25.862 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-ceph-common (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:25.876 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-cheroot. 2024-04-19T02:00:25.903 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-19T02:00:25.943 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-19T02:00:26.199 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-19T02:00:26.225 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-19T02:00:26.246 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-19T02:00:26.265 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-19T02:00:26.272 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-19T02:00:26.312 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-19T02:00:26.443 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ceph-base. 2024-04-19T02:00:26.470 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:26.528 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ceph-base (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:26.560 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-prettytable. 2024-04-19T02:00:26.582 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-19T02:00:26.586 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-19T02:00:26.609 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-19T02:00:26.618 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-19T02:00:26.648 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-19T02:00:26.845 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-rbd. 2024-04-19T02:00:26.872 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:26.912 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-rbd (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:26.951 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-19T02:00:26.978 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-19T02:00:27.018 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-19T02:00:27.106 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-19T02:00:27.133 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-19T02:00:27.173 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-19T02:00:27.257 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-19T02:00:27.283 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-19T02:00:27.321 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-tempora. 2024-04-19T02:00:27.323 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-19T02:00:27.347 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-19T02:00:27.387 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-19T02:00:27.442 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-cheroot. 2024-04-19T02:00:27.469 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-19T02:00:27.509 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-19T02:00:27.601 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-19T02:00:27.627 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-19T02:00:27.648 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-portend. 2024-04-19T02:00:27.664 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-19T02:00:27.667 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-19T02:00:27.698 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-19T02:00:27.820 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-19T02:00:27.847 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-19T02:00:27.887 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-19T02:00:27.970 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-19T02:00:27.996 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-19T02:00:28.001 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-19T02:00:28.030 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-19T02:00:28.030 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-19T02:00:28.076 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-19T02:00:28.198 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-19T02:00:28.224 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-19T02:00:28.264 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-19T02:00:28.349 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package lua5.1. 2024-04-19T02:00:28.364 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-19T02:00:28.379 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-19T02:00:28.406 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-19T02:00:28.407 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-19T02:00:28.445 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-19T02:00:28.567 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-19T02:00:28.593 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-19T02:00:28.634 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-19T02:00:28.710 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package lua-any. 2024-04-19T02:00:28.726 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-19T02:00:28.768 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-19T02:00:28.783 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-natsort. 2024-04-19T02:00:28.810 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-19T02:00:28.957 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-19T02:00:28.957 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-tempora. 2024-04-19T02:00:28.957 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-19T02:00:28.961 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-19T02:00:29.004 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package zip. 2024-04-19T02:00:29.019 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-19T02:00:29.054 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking zip (3.0-12build2) ... 2024-04-19T02:00:29.186 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-logutils. 2024-04-19T02:00:29.213 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-19T02:00:29.254 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-19T02:00:29.324 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package unzip. 2024-04-19T02:00:29.339 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-19T02:00:29.382 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-19T02:00:29.440 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-portend. 2024-04-19T02:00:29.466 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-19T02:00:29.506 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-19T02:00:29.622 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-19T02:00:29.649 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-19T02:00:29.697 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-19T02:00:29.770 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package luarocks. 2024-04-19T02:00:29.796 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-19T02:00:29.836 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-19T02:00:29.892 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-19T02:00:29.918 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-19T02:00:29.959 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-19T02:00:30.066 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-mako. 2024-04-19T02:00:30.093 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-19T02:00:30.141 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-19T02:00:30.245 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-19T02:00:30.261 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-19T02:00:30.295 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-19T02:00:30.352 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package librgw2. 2024-04-19T02:00:30.379 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../025-librgw2_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:30.444 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking librgw2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:30.511 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-19T02:00:30.538 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-19T02:00:30.578 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-19T02:00:30.641 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-natsort. 2024-04-19T02:00:30.667 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-19T02:00:30.707 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-19T02:00:30.864 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-19T02:00:30.894 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-19T02:00:30.930 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-19T02:00:30.971 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-rgw. 2024-04-19T02:00:30.986 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:30.993 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-logutils. 2024-04-19T02:00:31.019 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-19T02:00:31.029 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-rgw (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:31.060 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-19T02:00:31.283 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-webob. 2024-04-19T02:00:31.310 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-19T02:00:31.321 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-19T02:00:31.343 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-19T02:00:31.348 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-19T02:00:31.358 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-19T02:00:31.369 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-19T02:00:31.387 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-19T02:00:31.401 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-19T02:00:31.679 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libradosstriper1. 2024-04-19T02:00:31.690 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-mako. 2024-04-19T02:00:31.705 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:31.717 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-19T02:00:31.757 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-19T02:00:31.762 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libradosstriper1 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:31.820 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-waitress. 2024-04-19T02:00:31.847 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-19T02:00:31.898 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-19T02:00:32.060 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-19T02:00:32.086 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-19T02:00:32.127 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-19T02:00:32.182 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-common. 2024-04-19T02:00:32.208 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:32.249 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-common (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:32.272 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-tempita. 2024-04-19T02:00:32.299 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-19T02:00:32.347 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-19T02:00:32.396 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-19T02:00:32.411 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-19T02:00:32.446 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-19T02:00:32.667 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-paste. 2024-04-19T02:00:32.694 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-19T02:00:32.733 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-19T02:00:32.740 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-webob. 2024-04-19T02:00:32.767 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-19T02:00:32.807 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-19T02:00:33.053 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-19T02:00:33.080 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-19T02:00:33.120 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-19T02:00:33.160 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-waitress. 2024-04-19T02:00:33.187 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-19T02:00:33.230 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-19T02:00:33.757 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-19T02:00:33.783 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-19T02:00:34.173 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-19T02:00:34.288 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-base. 2024-04-19T02:00:34.305 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-tempita. 2024-04-19T02:00:34.315 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:34.332 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-19T02:00:34.374 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-base (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:34.488 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-19T02:00:34.735 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-webtest. 2024-04-19T02:00:34.762 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-19T02:00:34.775 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-paste. 2024-04-19T02:00:34.801 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-19T02:00:34.802 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-19T02:00:34.841 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-19T02:00:34.976 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-19T02:00:35.003 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-19T02:00:35.043 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-19T02:00:35.112 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pecan. 2024-04-19T02:00:35.139 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-19T02:00:35.161 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-19T02:00:35.179 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-19T02:00:35.188 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-19T02:00:35.228 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-19T02:00:35.313 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-cheroot. 2024-04-19T02:00:35.338 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-19T02:00:35.379 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-19T02:00:35.506 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-19T02:00:35.532 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-19T02:00:35.567 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-certifi. 2024-04-19T02:00:35.572 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-19T02:00:35.594 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-19T02:00:35.634 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-19T02:00:35.690 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-19T02:00:35.717 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-19T02:00:35.757 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-19T02:00:35.875 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-webtest. 2024-04-19T02:00:35.901 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-19T02:00:35.928 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-idna. 2024-04-19T02:00:35.941 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-19T02:00:35.955 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-19T02:00:35.995 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-19T02:00:36.060 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-19T02:00:36.086 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-19T02:00:36.127 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-19T02:00:36.245 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-pecan. 2024-04-19T02:00:36.271 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-19T02:00:36.311 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-19T02:00:36.314 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-urllib3. 2024-04-19T02:00:36.341 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-19T02:00:36.381 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-19T02:00:36.404 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-19T02:00:36.422 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-19T02:00:36.454 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-19T02:00:36.683 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-certifi. 2024-04-19T02:00:36.701 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-requests. 2024-04-19T02:00:36.709 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-19T02:00:36.724 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-tempora. 2024-04-19T02:00:36.728 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-19T02:00:36.749 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-19T02:00:36.750 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-19T02:00:36.767 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-19T02:00:36.790 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-19T02:00:37.044 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-idna. 2024-04-19T02:00:37.070 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-19T02:00:37.079 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-19T02:00:37.094 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-portend. 2024-04-19T02:00:37.106 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-19T02:00:37.110 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-19T02:00:37.120 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-19T02:00:37.145 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-19T02:00:37.160 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-19T02:00:37.380 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-urllib3. 2024-04-19T02:00:37.406 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-19T02:00:37.442 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-19T02:00:37.446 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-19T02:00:37.446 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-19T02:00:37.456 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-19T02:00:37.473 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-19T02:00:37.498 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:37.513 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-19T02:00:37.758 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-requests. 2024-04-19T02:00:37.774 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-19T02:00:37.785 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-19T02:00:37.801 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-19T02:00:37.824 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-19T02:00:37.841 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-19T02:00:37.861 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-19T02:00:37.877 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:37.920 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:38.136 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-19T02:00:38.162 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-19T02:00:38.186 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-natsort. 2024-04-19T02:00:38.202 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-19T02:00:38.211 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-19T02:00:38.239 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-19T02:00:38.253 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-19T02:00:38.266 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-19T02:00:38.306 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-19T02:00:38.472 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-19T02:00:38.488 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-19T02:00:38.522 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:38.564 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-logutils. 2024-04-19T02:00:38.575 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-openssl. 2024-04-19T02:00:38.590 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-19T02:00:38.602 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-19T02:00:38.631 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-19T02:00:38.633 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-19T02:00:38.902 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-19T02:00:38.926 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-19T02:00:38.928 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:38.953 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-19T02:00:38.962 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mgr. 2024-04-19T02:00:38.968 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:38.984 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-19T02:00:38.989 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:39.028 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mgr (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:39.265 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-19T02:00:39.270 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-mako. 2024-04-19T02:00:39.292 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-19T02:00:39.299 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-19T02:00:39.353 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-19T02:00:39.353 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-19T02:00:39.406 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mon. 2024-04-19T02:00:39.434 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:39.472 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mon (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:39.690 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-19T02:00:39.716 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-19T02:00:39.756 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-19T02:00:39.810 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-openssl. 2024-04-19T02:00:39.837 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-19T02:00:39.876 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-19T02:00:40.001 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-19T02:00:40.016 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-19T02:00:40.051 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-19T02:00:40.060 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-osd. 2024-04-19T02:00:40.087 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:40.126 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-osd (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:40.222 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ceph-mgr. 2024-04-19T02:00:40.249 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:40.288 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ceph-mgr (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:40.562 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-webob. 2024-04-19T02:00:40.588 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-19T02:00:40.629 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-19T02:00:40.642 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ceph-mon. 2024-04-19T02:00:40.669 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:40.700 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ceph-mon (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:40.948 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-waitress. 2024-04-19T02:00:40.975 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-19T02:00:41.018 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-19T02:00:41.204 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph. 2024-04-19T02:00:41.232 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../066-ceph_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:41.272 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:41.312 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ceph-osd. 2024-04-19T02:00:41.335 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-tempita. 2024-04-19T02:00:41.342 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:41.360 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-19T02:00:41.378 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ceph-osd (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:41.401 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-19T02:00:41.574 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-fuse. 2024-04-19T02:00:41.601 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:41.640 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-fuse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:41.679 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-paste. 2024-04-19T02:00:41.706 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-19T02:00:41.746 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-19T02:00:41.943 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mds. 2024-04-19T02:00:41.970 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:42.009 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mds (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:42.057 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-19T02:00:42.072 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-19T02:00:42.115 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-19T02:00:42.402 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-19T02:00:42.413 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package cephadm. 2024-04-19T02:00:42.428 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-19T02:00:42.440 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../069-cephadm_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:42.441 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ceph. 2024-04-19T02:00:42.468 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../066-ceph_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:42.468 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-19T02:00:42.479 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking cephadm (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:42.507 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ceph (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:42.722 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-webtest. 2024-04-19T02:00:42.748 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-19T02:00:42.778 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ceph-fuse. 2024-04-19T02:00:42.796 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-19T02:00:42.804 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:42.807 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-19T02:00:42.834 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-19T02:00:42.843 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ceph-fuse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:42.873 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-19T02:00:43.108 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-pecan. 2024-04-19T02:00:43.135 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-19T02:00:43.174 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-19T02:00:43.201 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-19T02:00:43.217 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-19T02:00:43.247 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ceph-mds. 2024-04-19T02:00:43.259 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-19T02:00:43.274 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:43.314 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ceph-mds (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:43.521 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-certifi. 2024-04-19T02:00:43.547 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-19T02:00:43.587 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-19T02:00:43.726 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package cephadm. 2024-04-19T02:00:43.752 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../069-cephadm_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:43.792 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking cephadm (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:43.916 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-idna. 2024-04-19T02:00:43.942 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-19T02:00:43.982 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-19T02:00:44.078 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-19T02:00:44.106 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-19T02:00:44.145 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-19T02:00:44.235 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-urllib3. 2024-04-19T02:00:44.261 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-19T02:00:44.293 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-19T02:00:44.465 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-19T02:00:44.480 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-19T02:00:44.514 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-19T02:00:44.522 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-babel. 2024-04-19T02:00:44.550 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-19T02:00:44.589 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-requests. 2024-04-19T02:00:44.589 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-19T02:00:44.615 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-19T02:00:44.646 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-19T02:00:44.924 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-19T02:00:44.942 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-jinja2. 2024-04-19T02:00:44.951 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-19T02:00:44.969 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-19T02:00:44.991 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-19T02:00:45.008 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-19T02:00:45.253 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-19T02:00:45.279 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-19T02:00:45.311 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:45.370 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-19T02:00:45.397 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-19T02:00:45.436 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:45.512 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-babel. 2024-04-19T02:00:45.539 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-19T02:00:45.578 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-19T02:00:45.716 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-19T02:00:45.743 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:45.773 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-19T02:00:45.782 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:45.800 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-19T02:00:45.840 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-19T02:00:45.915 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-jinja2. 2024-04-19T02:00:45.942 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-19T02:00:45.982 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-19T02:00:46.044 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-19T02:00:46.071 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-19T02:00:46.102 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-19T02:00:46.117 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-routes. 2024-04-19T02:00:46.144 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-19T02:00:46.184 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-19T02:00:46.327 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-19T02:00:46.354 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-19T02:00:46.388 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-openssl. 2024-04-19T02:00:46.393 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:46.415 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-19T02:00:46.455 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-19T02:00:46.512 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-19T02:00:46.528 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-19T02:00:46.562 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:46.739 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-19T02:00:46.766 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-19T02:00:46.791 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-mgr. 2024-04-19T02:00:46.805 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-19T02:00:46.818 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:46.883 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-mgr (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:47.059 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-routes. 2024-04-19T02:00:47.085 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-19T02:00:47.125 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-19T02:00:47.278 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-mon. 2024-04-19T02:00:47.305 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:47.344 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-mon (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:47.437 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-19T02:00:47.464 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-19T02:00:47.495 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:47.825 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-19T02:00:47.842 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-19T02:00:47.883 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-19T02:00:47.923 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-osd. 2024-04-19T02:00:47.949 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:47.989 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-osd (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:48.387 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-joblib. 2024-04-19T02:00:48.414 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-19T02:00:48.454 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-19T02:00:48.807 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-19T02:00:48.809 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-19T02:00:48.825 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-19T02:00:48.835 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-19T02:00:48.868 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-19T02:00:48.873 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-19T02:00:49.043 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph. 2024-04-19T02:00:49.069 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../066-ceph_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:49.118 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:49.143 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-sklearn. 2024-04-19T02:00:49.170 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-19T02:00:49.209 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-19T02:00:49.388 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-joblib. 2024-04-19T02:00:49.415 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-19T02:00:49.437 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-fuse. 2024-04-19T02:00:49.454 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-19T02:00:49.464 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:49.504 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-fuse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:49.801 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-19T02:00:49.824 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-mds. 2024-04-19T02:00:49.828 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-19T02:00:49.840 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:49.867 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-19T02:00:49.876 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-19T02:00:49.882 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-mds (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:49.893 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-19T02:00:50.020 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:50.269 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package cephadm. 2024-04-19T02:00:50.296 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../069-cephadm_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:50.320 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-sklearn. 2024-04-19T02:00:50.335 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking cephadm (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:50.347 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-19T02:00:50.386 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-19T02:00:50.655 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-19T02:00:50.681 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-19T02:00:50.854 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-19T02:00:51.007 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-cachetools. 2024-04-19T02:00:51.035 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-19T02:00:51.062 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-19T02:00:51.073 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-19T02:00:51.088 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-19T02:00:51.182 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-19T02:00:51.198 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-19T02:00:51.215 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:51.240 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-19T02:00:51.360 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-19T02:00:51.387 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-19T02:00:51.426 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-19T02:00:51.763 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-19T02:00:51.790 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-19T02:00:51.830 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-19T02:00:52.124 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-rsa. 2024-04-19T02:00:52.151 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-19T02:00:52.190 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-19T02:00:52.221 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-babel. 2024-04-19T02:00:52.228 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-cachetools. 2024-04-19T02:00:52.248 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-19T02:00:52.255 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-19T02:00:52.287 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-19T02:00:52.294 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-19T02:00:52.502 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-google-auth. 2024-04-19T02:00:52.530 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-19T02:00:52.569 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-19T02:00:52.572 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-19T02:00:52.600 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-19T02:00:52.607 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-jinja2. 2024-04-19T02:00:52.634 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-19T02:00:52.639 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-19T02:00:52.673 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-19T02:00:52.839 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-19T02:00:52.866 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-19T02:00:52.897 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-19T02:00:52.959 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-19T02:00:52.987 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-19T02:00:53.018 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-19T02:00:53.026 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-19T02:00:53.045 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-19T02:00:53.085 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:53.175 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-websocket. 2024-04-19T02:00:53.202 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-19T02:00:53.241 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-19T02:00:53.313 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-rsa. 2024-04-19T02:00:53.340 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-19T02:00:53.379 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-19T02:00:53.430 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-19T02:00:53.456 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-19T02:00:53.496 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-19T02:00:53.570 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-19T02:00:53.597 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-19T02:00:53.636 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-19T02:00:53.700 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-google-auth. 2024-04-19T02:00:53.727 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-19T02:00:53.767 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-19T02:00:53.799 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-routes. 2024-04-19T02:00:53.826 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-19T02:00:53.866 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-19T02:00:54.087 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-19T02:00:54.114 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-19T02:00:54.153 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-19T02:00:54.203 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-19T02:00:54.229 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-19T02:00:54.269 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:54.359 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-19T02:00:54.389 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-19T02:00:54.432 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-websocket. 2024-04-19T02:00:54.448 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-19T02:00:54.481 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-19T02:00:54.575 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:54.761 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-19T02:00:54.788 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-19T02:00:54.827 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-19T02:00:54.911 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-19T02:00:54.939 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-19T02:00:54.977 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-19T02:00:55.306 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-19T02:00:55.334 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-19T02:00:55.364 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-19T02:00:55.617 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package jq. 2024-04-19T02:00:55.644 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-19T02:00:55.649 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-19T02:00:55.650 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-19T02:00:55.665 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-19T02:00:55.678 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-19T02:00:55.683 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-19T02:00:55.707 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-19T02:00:55.731 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:55.936 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package socat. 2024-04-19T02:00:55.952 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-19T02:00:55.994 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-19T02:00:56.028 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-19T02:00:56.055 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-19T02:00:56.094 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-19T02:00:56.211 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-joblib. 2024-04-19T02:00:56.238 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-19T02:00:56.278 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-19T02:00:56.349 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package xmlstarlet. 2024-04-19T02:00:56.366 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-19T02:00:56.406 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-19T02:00:56.416 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-19T02:00:56.433 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-19T02:00:56.481 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-19T02:00:56.648 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-19T02:00:56.675 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-19T02:00:56.714 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-19T02:00:56.751 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package jq. 2024-04-19T02:00:56.753 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-test. 2024-04-19T02:00:56.770 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:56.778 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-19T02:00:56.811 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-test (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:56.817 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-19T02:00:57.000 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-sklearn. 2024-04-19T02:00:57.027 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-19T02:00:57.062 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package socat. 2024-04-19T02:00:57.067 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-19T02:00:57.078 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-19T02:00:57.112 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-19T02:00:57.399 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package xmlstarlet. 2024-04-19T02:00:57.415 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-19T02:00:57.449 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-19T02:00:57.804 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ceph-test. 2024-04-19T02:00:57.817 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-19T02:00:57.819 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:57.845 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-19T02:00:57.861 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ceph-test (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:57.970 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:58.915 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-cachetools. 2024-04-19T02:00:58.942 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-19T02:00:58.981 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-19T02:00:59.182 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ceph-volume. 2024-04-19T02:00:59.210 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-19T02:00:59.258 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ceph-volume (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:59.309 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-19T02:00:59.336 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-19T02:00:59.376 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-19T02:00:59.611 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-19T02:00:59.628 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:00:59.669 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libcephfs-dev (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:00:59.730 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-19T02:00:59.757 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-19T02:00:59.796 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-19T02:00:59.997 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-19T02:01:00.025 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-19T02:01:00.073 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-19T02:01:00.107 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-rsa. 2024-04-19T02:01:00.134 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-19T02:01:00.174 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-19T02:01:00.418 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-19T02:01:00.436 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-google-auth. 2024-04-19T02:01:00.446 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-19T02:01:00.463 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-19T02:01:00.484 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-19T02:01:00.494 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-19T02:01:00.617 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ceph-volume. 2024-04-19T02:01:00.644 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-19T02:01:00.692 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ceph-volume (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:00.806 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-19T02:01:00.830 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package nvme-cli. 2024-04-19T02:01:00.833 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-19T02:01:00.858 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-19T02:01:00.872 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-19T02:01:00.896 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-19T02:01:01.030 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-19T02:01:01.057 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:01:01.104 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libcephfs-dev (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:01.318 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package pkg-config. 2024-04-19T02:01:01.335 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-19T02:01:01.376 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-19T02:01:01.401 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-websocket. 2024-04-19T02:01:01.428 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-19T02:01:01.441 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-19T02:01:01.467 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-19T02:01:01.468 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-19T02:01:01.507 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-19T02:01:01.630 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-19T02:01:01.658 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-19T02:01:01.696 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-19T02:01:01.771 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-19T02:01:01.798 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-19T02:01:01.829 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-19T02:01:01.879 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-19T02:01:01.907 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-19T02:01:01.946 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-19T02:01:02.125 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-19T02:01:02.152 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-19T02:01:02.191 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-19T02:01:02.384 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package nvme-cli. 2024-04-19T02:01:02.411 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-19T02:01:02.443 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-19T02:01:02.458 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-19T02:01:02.459 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-19T02:01:02.460 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pastescript. 2024-04-19T02:01:02.477 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-19T02:01:02.501 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:02.518 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-19T02:01:02.796 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-19T02:01:02.823 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-19T02:01:02.838 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pluggy. 2024-04-19T02:01:02.848 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package pkg-config. 2024-04-19T02:01:02.862 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-19T02:01:02.866 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-19T02:01:02.875 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-19T02:01:02.905 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-19T02:01:02.914 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-19T02:01:03.151 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-19T02:01:03.165 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-19T02:01:03.167 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-19T02:01:03.192 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-19T02:01:03.209 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-19T02:01:03.225 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-psutil. 2024-04-19T02:01:03.252 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-19T02:01:03.257 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-19T02:01:03.291 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-19T02:01:03.510 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package jq. 2024-04-19T02:01:03.537 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-19T02:01:03.576 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-19T02:01:03.604 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-19T02:01:03.632 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-19T02:01:03.636 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-py. 2024-04-19T02:01:03.664 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-19T02:01:03.673 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-19T02:01:03.710 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-19T02:01:03.838 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package socat. 2024-04-19T02:01:03.865 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-19T02:01:03.904 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-19T02:01:04.007 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-pastescript. 2024-04-19T02:01:04.034 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-19T02:01:04.074 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-19T02:01:04.097 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pygments. 2024-04-19T02:01:04.125 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-19T02:01:04.163 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-19T02:01:04.182 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package xmlstarlet. 2024-04-19T02:01:04.198 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-19T02:01:04.240 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-19T02:01:04.394 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-pluggy. 2024-04-19T02:01:04.422 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-19T02:01:04.461 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-19T02:01:04.586 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-test. 2024-04-19T02:01:04.602 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:01:04.618 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-19T02:01:04.636 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-test (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:04.646 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-19T02:01:04.684 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-19T02:01:04.747 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-psutil. 2024-04-19T02:01:04.775 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-19T02:01:04.805 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-19T02:01:04.937 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-toml. 2024-04-19T02:01:04.965 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-19T02:01:04.995 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-19T02:01:05.151 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-py. 2024-04-19T02:01:05.178 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-19T02:01:05.225 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-19T02:01:05.448 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-pytest. 2024-04-19T02:01:05.477 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-19T02:01:05.512 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-pygments. 2024-04-19T02:01:05.514 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-19T02:01:05.540 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-19T02:01:05.579 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-19T02:01:05.802 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package python3-simplejson. 2024-04-19T02:01:05.830 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-19T02:01:05.859 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-19T02:01:06.050 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-19T02:01:06.078 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-19T02:01:06.108 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-19T02:01:06.129 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-19T02:01:06.146 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-19T02:01:06.187 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-19T02:01:06.395 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-toml. 2024-04-19T02:01:06.422 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-19T02:01:06.461 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-19T02:01:06.764 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-pytest. 2024-04-19T02:01:06.792 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-19T02:01:06.808 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-volume. 2024-04-19T02:01:06.831 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-19T02:01:06.835 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2369-g1a6a3e74-1jammy_all.deb ... 2024-04-19T02:01:06.874 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-volume (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:06.875 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package radosgw. 2024-04-19T02:01:06.904 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../116-radosgw_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:01:06.941 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking radosgw (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:07.152 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-simplejson. 2024-04-19T02:01:07.170 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-19T02:01:07.179 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-19T02:01:07.197 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:01:07.218 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-19T02:01:07.236 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libcephfs-dev (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:07.488 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-19T02:01:07.504 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-19T02:01:07.546 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-19T02:01:07.564 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-19T02:01:07.592 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-19T02:01:07.630 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-19T02:01:07.761 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package rbd-fuse. 2024-04-19T02:01:07.790 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:01:07.827 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking rbd-fuse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:07.960 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-19T02:01:07.988 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-19T02:01:08.026 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-19T02:01:08.149 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-19T02:01:08.318 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package radosgw. 2024-04-19T02:01:08.339 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package nvme-cli. 2024-04-19T02:01:08.346 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../116-radosgw_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:01:08.369 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-19T02:01:08.384 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking radosgw (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:08.406 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-19T02:01:08.457 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-19T02:01:08.558 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-19T02:01:08.803 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package pkg-config. 2024-04-19T02:01:08.807 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-19T02:01:08.830 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-19T02:01:08.869 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-19T02:01:09.114 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-19T02:01:09.130 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-19T02:01:09.147 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-19T02:01:09.172 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-19T02:01:09.179 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package rbd-fuse. 2024-04-19T02:01:09.195 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:01:09.237 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking rbd-fuse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:09.550 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-19T02:01:09.606 INFO:teuthology.orchestra.run.smithi059.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-19T02:01:09.625 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-19T02:01:09.653 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-19T02:01:09.677 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up cephadm (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:09.691 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-19T02:01:09.801 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-19T02:01:09.927 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-19T02:01:09.994 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-pastescript. 2024-04-19T02:01:10.022 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-19T02:01:10.061 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-19T02:01:10.187 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-19T02:01:10.325 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-pluggy. 2024-04-19T02:01:10.352 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-19T02:01:10.383 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-19T02:01:10.444 INFO:teuthology.orchestra.run.smithi059.stdout:Adding system user cephadm....done 2024-04-19T02:01:10.499 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-19T02:01:10.600 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-19T02:01:10.694 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-psutil. 2024-04-19T02:01:10.722 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-19T02:01:10.760 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-19T02:01:10.876 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-19T02:01:10.903 INFO:teuthology.orchestra.run.smithi179.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-19T02:01:10.963 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up cephadm (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:11.089 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-py. 2024-04-19T02:01:11.117 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-19T02:01:11.155 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-19T02:01:11.185 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-19T02:01:11.311 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-19T02:01:11.488 INFO:teuthology.orchestra.run.smithi179.stdout:Adding system user cephadm....done 2024-04-19T02:01:11.492 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-pygments. 2024-04-19T02:01:11.520 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-19T02:01:11.558 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-19T02:01:11.571 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-19T02:01:11.603 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-19T02:01:11.830 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-19T02:01:11.879 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-19T02:01:11.948 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-19T02:01:12.013 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-19T02:01:12.040 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-19T02:01:12.079 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-19T02:01:12.139 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-19T02:01:12.266 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-19T02:01:12.267 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-19T02:01:12.391 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-toml. 2024-04-19T02:01:12.418 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-19T02:01:12.457 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-19T02:01:12.517 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-19T02:01:12.627 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-19T02:01:12.718 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-pytest. 2024-04-19T02:01:12.734 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-19T02:01:12.776 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-19T02:01:12.777 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-19T02:01:12.903 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-19T02:01:12.903 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-19T02:01:13.121 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-simplejson. 2024-04-19T02:01:13.149 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-19T02:01:13.188 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-19T02:01:13.188 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-19T02:01:13.204 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-19T02:01:13.414 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-19T02:01:13.466 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-19T02:01:13.494 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-19T02:01:13.532 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-19T02:01:13.565 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-19T02:01:13.657 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-19T02:01:13.825 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-19T02:01:13.916 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-19T02:01:14.085 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-19T02:01:14.209 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-ceph-argparse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:14.245 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package radosgw. 2024-04-19T02:01:14.273 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../116-radosgw_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:01:14.311 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking radosgw (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:14.353 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-19T02:01:14.460 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-19T02:01:14.586 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-19T02:01:14.612 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-19T02:01:14.712 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-19T02:01:14.821 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-19T02:01:14.864 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-19T02:01:15.064 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-19T02:01:15.098 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-ceph-argparse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:15.147 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package rbd-fuse. 2024-04-19T02:01:15.164 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2369-g1a6a3e74-1jammy_amd64.deb ... 2024-04-19T02:01:15.206 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking rbd-fuse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:15.375 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-19T02:01:15.445 INFO:teuthology.orchestra.run.smithi059.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-19T02:01:15.479 INFO:teuthology.orchestra.run.smithi059.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-19T02:01:15.510 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-19T02:01:15.535 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-19T02:01:15.601 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-19T02:01:15.643 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-19T02:01:15.727 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-19T02:01:15.777 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-19T02:01:15.853 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-19T02:01:15.904 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-19T02:01:16.020 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-19T02:01:16.113 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-19T02:01:16.146 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-19T02:01:16.160 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-19T02:01:16.397 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-19T02:01:16.494 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-19T02:01:16.524 INFO:teuthology.orchestra.run.smithi179.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-19T02:01:16.570 INFO:teuthology.orchestra.run.smithi179.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-19T02:01:16.633 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-19T02:01:16.648 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-19T02:01:16.743 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-19T02:01:16.957 INFO:teuthology.orchestra.run.smithi081.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-19T02:01:16.974 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up zip (3.0-12build2) ... 2024-04-19T02:01:17.065 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up cephadm (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:17.085 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-19T02:01:17.100 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-19T02:01:17.212 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-19T02:01:17.471 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-19T02:01:17.731 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-19T02:01:17.769 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-19T02:01:17.924 INFO:teuthology.orchestra.run.smithi081.stdout:Adding system user cephadm....done 2024-04-19T02:01:18.024 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up zip (3.0-12build2) ... 2024-04-19T02:01:18.028 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-19T02:01:18.080 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-19T02:01:18.159 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-19T02:01:18.171 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-19T02:01:18.272 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-19T02:01:18.348 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-19T02:01:18.565 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-19T02:01:18.599 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-19T02:01:18.717 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-19T02:01:18.827 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-19T02:01:18.968 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-19T02:01:18.974 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-19T02:01:19.087 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-19T02:01:19.213 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-19T02:01:19.226 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-19T02:01:19.227 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-19T02:01:19.339 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-19T02:01:19.354 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-19T02:01:19.610 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-19T02:01:19.632 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-19T02:01:19.664 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-19T02:01:19.903 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-19T02:01:20.024 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-19T02:01:20.033 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-19T02:01:20.254 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-19T02:01:20.276 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-19T02:01:20.284 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-19T02:01:20.514 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-19T02:01:20.560 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-19T02:01:20.623 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-ceph-common (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:20.686 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-19T02:01:20.820 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-19T02:01:20.916 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-idna (3.3-1) ... 2024-04-19T02:01:20.963 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-19T02:01:21.080 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-19T02:01:21.234 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-19T02:01:21.347 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-19T02:01:21.440 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-19T02:01:21.682 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-ceph-argparse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:21.693 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-19T02:01:21.819 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-ceph-common (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:21.933 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-19T02:01:22.061 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-19T02:01:22.112 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-idna (3.3-1) ... 2024-04-19T02:01:22.185 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-19T02:01:22.311 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-19T02:01:22.438 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-19T02:01:22.481 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-19T02:01:22.570 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-19T02:01:22.716 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T02:01:22.850 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-19T02:01:22.950 INFO:teuthology.orchestra.run.smithi081.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-19T02:01:22.984 INFO:teuthology.orchestra.run.smithi081.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-19T02:01:23.041 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-19T02:01:23.118 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-19T02:01:23.142 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-19T02:01:23.244 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-19T02:01:23.370 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-19T02:01:23.502 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-19T02:01:23.603 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-19T02:01:23.627 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-19T02:01:23.671 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-19T02:01:23.813 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T02:01:23.914 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-19T02:01:23.923 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-19T02:01:23.928 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-19T02:01:24.174 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T02:01:24.190 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-19T02:01:24.204 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-19T02:01:24.300 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-19T02:01:24.317 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-19T02:01:24.460 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-19T02:01:24.506 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up zip (3.0-12build2) ... 2024-04-19T02:01:24.593 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-19T02:01:24.632 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-19T02:01:24.736 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-19T02:01:24.836 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-19T02:01:24.987 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-19T02:01:25.112 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-19T02:01:25.247 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T02:01:25.301 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-19T02:01:25.365 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-19T02:01:25.388 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-19T02:01:25.560 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-19T02:01:25.647 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-19T02:01:25.666 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-19T02:01:25.686 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-19T02:01:25.837 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-19T02:01:25.915 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-19T02:01:25.918 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-19T02:01:26.033 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-19T02:01:26.147 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-19T02:01:26.177 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-19T02:01:26.309 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-19T02:01:26.410 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-19T02:01:26.437 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-19T02:01:26.532 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-19T02:01:26.664 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-19T02:01:26.711 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-19T02:01:26.783 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-19T02:01:26.923 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-19T02:01:27.020 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-19T02:01:27.049 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-19T02:01:27.159 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-19T02:01:27.271 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-19T02:01:27.342 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-19T02:01:27.372 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-19T02:01:27.460 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-19T02:01:27.490 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-19T02:01:27.569 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-19T02:01:27.579 INFO:teuthology.orchestra.run.smithi059.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-19T02:01:27.728 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-19T02:01:27.833 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-19T02:01:27.929 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-19T02:01:27.959 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-19T02:01:28.021 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-19T02:01:28.205 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-19T02:01:28.272 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-19T02:01:28.348 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-ceph-common (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:28.377 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-19T02:01:28.381 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-19T02:01:28.532 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-19T02:01:28.622 INFO:teuthology.orchestra.run.smithi179.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-19T02:01:28.645 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-19T02:01:28.657 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-idna (3.3-1) ... 2024-04-19T02:01:28.763 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-19T02:01:28.859 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-19T02:01:28.975 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-19T02:01:28.977 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-19T02:01:29.006 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T02:01:29.140 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-19T02:01:29.362 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-19T02:01:29.466 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-19T02:01:29.597 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-19T02:01:29.698 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-19T02:01:29.792 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-19T02:01:29.957 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T02:01:30.077 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-19T02:01:30.083 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-19T02:01:30.140 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-19T02:01:30.203 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-19T02:01:30.383 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T02:01:30.401 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-19T02:01:30.496 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-19T02:01:30.517 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-19T02:01:30.695 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-19T02:01:30.763 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-19T02:01:30.785 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-19T02:01:30.911 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-19T02:01:30.971 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-19T02:01:31.062 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-19T02:01:31.072 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-19T02:01:31.108 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-19T02:01:31.365 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-19T02:01:31.381 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-19T02:01:31.493 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-19T02:01:31.624 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-19T02:01:31.649 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-19T02:01:31.787 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-19T02:01:31.875 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T02:01:32.012 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-19T02:01:32.025 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-19T02:01:32.131 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-19T02:01:32.265 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up librados2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:32.327 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-19T02:01:32.380 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-19T02:01:32.391 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:32.542 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-19T02:01:32.586 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-19T02:01:32.666 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-19T02:01:32.863 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-19T02:01:32.945 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libcephfs2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:33.002 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-19T02:01:33.096 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libradosstriper1 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:33.106 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-19T02:01:33.111 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up librados2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:33.222 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-19T02:01:33.238 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:33.340 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-19T02:01:33.364 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-19T02:01:33.516 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up librbd1 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:33.599 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-19T02:01:33.658 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mgr-modules-core (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:33.700 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libcephfs2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:33.726 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-19T02:01:33.809 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-fuse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:33.835 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libradosstriper1 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:33.978 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-19T02:01:34.010 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-19T02:01:34.044 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-19T02:01:34.044 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-19T02:01:34.136 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-19T02:01:34.288 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up librbd1 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:34.439 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ceph-mgr-modules-core (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:34.445 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-19T02:01:34.528 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libcephfs-dev (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:34.590 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ceph-fuse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:34.679 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-rados (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:34.747 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-19T02:01:34.800 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-19T02:01:34.801 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-19T02:01:35.015 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-19T02:01:35.141 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-19T02:01:35.164 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up librgw2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:35.265 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-19T02:01:35.266 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libcephfs-dev (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:35.275 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-19T02:01:35.364 INFO:teuthology.orchestra.run.smithi081.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-19T02:01:35.393 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-rados (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:35.536 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up librgw2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:35.602 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-19T02:01:35.687 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-19T02:01:35.728 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-19T02:01:36.130 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-19T02:01:36.406 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-19T02:01:36.524 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-19T02:01:36.527 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-rbd (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:36.653 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up rbd-fuse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:36.779 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-rgw (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:36.791 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T02:01:36.888 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up python3-cephfs (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:36.917 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-19T02:01:36.933 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-rbd (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:36.989 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-common (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:37.059 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up rbd-fuse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:37.186 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-rgw (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:37.243 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-19T02:01:37.337 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-cephfs (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:37.488 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ceph-common (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:37.549 INFO:teuthology.orchestra.run.smithi059.stdout:Adding group ceph....done 2024-04-19T02:01:37.553 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-19T02:01:37.846 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-19T02:01:37.901 INFO:teuthology.orchestra.run.smithi059.stdout:Adding system user ceph....done 2024-04-19T02:01:37.963 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-19T02:01:38.043 INFO:teuthology.orchestra.run.smithi059.stdout:Setting system user ceph properties....done 2024-04-19T02:01:38.058 INFO:teuthology.orchestra.run.smithi059.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-19T02:01:38.081 INFO:teuthology.orchestra.run.smithi179.stdout:Adding group ceph....done 2024-04-19T02:01:38.198 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-19T02:01:38.223 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-19T02:01:38.492 INFO:teuthology.orchestra.run.smithi179.stdout:Adding system user ceph....done 2024-04-19T02:01:38.499 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-19T02:01:38.518 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-19T02:01:38.703 INFO:teuthology.orchestra.run.smithi179.stdout:Setting system user ceph properties....done 2024-04-19T02:01:38.717 INFO:teuthology.orchestra.run.smithi179.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-19T02:01:38.853 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-19T02:01:38.854 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-19T02:01:38.982 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-test (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:39.108 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up radosgw (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:39.206 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-19T02:01:39.271 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-19T02:01:39.540 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-19T02:01:39.683 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ceph-test (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:39.749 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-19T02:01:39.749 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-19T02:01:39.809 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up radosgw (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:39.867 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-19T02:01:39.993 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up librados2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:40.144 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:40.243 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-base (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:40.270 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-19T02:01:40.283 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-19T02:01:40.283 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-19T02:01:40.590 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libcephfs2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:40.640 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-19T02:01:40.709 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libradosstriper1 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:40.745 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ceph-base (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:40.834 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-19T02:01:41.086 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-19T02:01:41.110 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up librbd1 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:41.128 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mds (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:41.253 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-mgr-modules-core (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:41.381 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-19T02:01:41.381 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-19T02:01:41.404 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-fuse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:41.555 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ceph-mds (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:41.635 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-19T02:01:41.635 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-19T02:01:41.785 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-19T02:01:41.785 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-19T02:01:41.888 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mgr (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:42.097 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-19T02:01:42.097 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-19T02:01:42.147 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libcephfs-dev (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:42.241 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ceph-mgr (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:42.265 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-rados (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:42.383 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up librgw2 (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:42.465 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-19T02:01:42.465 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-19T02:01:42.509 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-19T02:01:42.573 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-osd (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:42.858 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-19T02:01:42.859 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-19T02:01:42.942 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ceph-osd (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:43.230 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-19T02:01:43.230 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-19T02:01:43.350 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:43.476 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:43.627 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mon (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:43.678 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:43.729 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-rbd (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:43.829 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:43.846 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-19T02:01:43.847 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-19T02:01:43.847 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up rbd-fuse (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:43.948 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-rgw (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:43.997 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ceph-mon (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:44.058 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-cephfs (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:44.200 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-common (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:44.210 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-19T02:01:44.210 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-19T02:01:44.361 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mgr-cephadm (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:44.521 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:44.630 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-mgr-dashboard (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:44.649 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ceph-mgr-cephadm (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:44.756 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ceph-volume (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:44.792 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ceph (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:44.793 INFO:teuthology.orchestra.run.smithi081.stdout:Adding group ceph....done 2024-04-19T02:01:44.943 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ceph-mgr-dashboard (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:45.094 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ceph-volume (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:45.143 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-19T02:01:45.162 INFO:teuthology.orchestra.run.smithi081.stdout:Adding system user ceph....done 2024-04-19T02:01:45.243 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-19T02:01:45.288 INFO:teuthology.orchestra.run.smithi081.stdout:Setting system user ceph properties....done 2024-04-19T02:01:45.306 INFO:teuthology.orchestra.run.smithi081.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-19T02:01:45.396 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-19T02:01:45.430 INFO:teuthology.orchestra.run.smithi179.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-19T02:01:45.445 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-19T02:01:45.548 INFO:teuthology.orchestra.run.smithi179.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-19T02:01:45.701 INFO:teuthology.orchestra.run.smithi179.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-19T02:01:45.788 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-19T02:01:46.269 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-test (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:46.446 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up radosgw (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:46.563 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-19T02:01:46.745 INFO:teuthology.orchestra.run.smithi179.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-19T02:01:46.966 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-19T02:01:46.967 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-19T02:01:47.230 INFO:teuthology.orchestra.run.smithi059.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-19T02:01:47.236 DEBUG:teuthology.parallel:result is None 2024-04-19T02:01:47.381 INFO:teuthology.orchestra.run.smithi179.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-19T02:01:47.386 DEBUG:teuthology.parallel:result is None 2024-04-19T02:01:47.495 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-base (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:48.031 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-19T02:01:48.535 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-mds (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:48.772 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-19T02:01:48.772 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-19T02:01:49.236 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-mgr (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:49.454 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-19T02:01:49.454 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-19T02:01:49.946 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-osd (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:50.214 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-19T02:01:50.214 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-19T02:01:50.739 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:50.890 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:51.042 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-mon (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:51.255 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-19T02:01:51.255 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-19T02:01:51.751 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-mgr-cephadm (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:51.902 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:52.028 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-mgr-dashboard (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:52.255 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-volume (19.0.0-2369-g1a6a3e74-1jammy) ... 2024-04-19T02:01:52.608 INFO:teuthology.orchestra.run.smithi081.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-19T02:01:52.709 INFO:teuthology.orchestra.run.smithi081.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-19T02:01:52.861 INFO:teuthology.orchestra.run.smithi081.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-19T02:01:53.629 INFO:teuthology.orchestra.run.smithi081.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-19T02:01:54.253 INFO:teuthology.orchestra.run.smithi081.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-19T02:01:54.258 DEBUG:teuthology.parallel:result is None 2024-04-19T02:01:54.259 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1a6a3e742fd81c6d70be70552499bff767652c32 2024-04-19T02:01:54.489 DEBUG:teuthology.orchestra.run.smithi059:> dpkg-query -W -f '${Version}' ceph 2024-04-19T02:01:54.509 INFO:teuthology.orchestra.run.smithi059.stdout:19.0.0-2369-g1a6a3e74-1jammy 2024-04-19T02:01:54.509 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2369-g1a6a3e74-1jammy 2024-04-19T02:01:54.509 INFO:teuthology.task.install:The correct ceph version 19.0.0-2369-g1a6a3e74-1jammy is installed. 2024-04-19T02:01:54.511 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1a6a3e742fd81c6d70be70552499bff767652c32 2024-04-19T02:01:54.664 DEBUG:teuthology.orchestra.run.smithi081:> dpkg-query -W -f '${Version}' ceph 2024-04-19T02:01:54.683 INFO:teuthology.orchestra.run.smithi081.stdout:19.0.0-2369-g1a6a3e74-1jammy 2024-04-19T02:01:54.683 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2369-g1a6a3e74-1jammy 2024-04-19T02:01:54.683 INFO:teuthology.task.install:The correct ceph version 19.0.0-2369-g1a6a3e74-1jammy is installed. 2024-04-19T02:01:54.685 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1a6a3e742fd81c6d70be70552499bff767652c32 2024-04-19T02:01:54.903 DEBUG:teuthology.orchestra.run.smithi179:> dpkg-query -W -f '${Version}' ceph 2024-04-19T02:01:54.922 INFO:teuthology.orchestra.run.smithi179.stdout:19.0.0-2369-g1a6a3e74-1jammy 2024-04-19T02:01:54.922 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2369-g1a6a3e74-1jammy 2024-04-19T02:01:54.922 INFO:teuthology.task.install:The correct ceph version 19.0.0-2369-g1a6a3e74-1jammy is installed. 2024-04-19T02:01:54.924 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-19T02:01:54.924 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-19T02:01:54.924 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-19T02:01:54.939 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-19T02:01:54.939 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-19T02:01:54.953 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-04-19T02:01:54.953 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-19T02:01:54.978 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-19T02:01:54.979 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-19T02:01:54.979 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/daemon-helper 2024-04-19T02:01:54.994 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-19T02:01:55.050 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-19T02:01:55.050 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/daemon-helper 2024-04-19T02:01:55.064 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-19T02:01:55.118 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-04-19T02:01:55.118 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/daemon-helper 2024-04-19T02:01:55.131 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-19T02:01:55.185 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-19T02:01:55.185 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-19T02:01:55.185 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-19T02:01:55.198 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-19T02:01:55.254 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-19T02:01:55.254 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-19T02:01:55.268 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-19T02:01:55.321 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-04-19T02:01:55.321 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-19T02:01:55.334 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-19T02:01:55.389 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-19T02:01:55.389 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-19T02:01:55.389 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/stdin-killer 2024-04-19T02:01:55.402 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-19T02:01:55.458 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-19T02:01:55.458 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/stdin-killer 2024-04-19T02:01:55.471 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-19T02:01:55.526 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-04-19T02:01:55.526 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/stdin-killer 2024-04-19T02:01:55.539 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-19T02:01:55.593 INFO:teuthology.run_tasks:Running task ceph... 2024-04-19T02:01:55.680 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-19T02:01:55.680 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 777 /var/log/ceph 2024-04-19T02:01:55.682 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 777 /var/log/ceph 2024-04-19T02:01:55.685 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 777 /var/log/ceph 2024-04-19T02:01:55.698 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-19T02:01:55.699 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-19T02:01:55.739 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-19T02:01:55.741 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-19T02:01:55.753 INFO:tasks.ceph:Creating extra log directories... 2024-04-19T02:01:55.753 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-19T02:01:55.795 DEBUG:teuthology.orchestra.run.smithi081:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-19T02:01:55.798 DEBUG:teuthology.orchestra.run.smithi179:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-19T02:01:55.821 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-19T02:01:55.821 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-19T02:01:55.821 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662749', 'branch': 'squid', 'description': 'rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7662749', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': '1a6a3e742fd81c6d70be70552499bff767652c32'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1a6a3e742fd81c6d70be70552499bff767652c32'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'd0bf63dd179c05751e4be49477389aa290c73089'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 101, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'mon.a'], ['osd.4', 'osd.5', 'osd.6', 'osd.7', 'mon.b', 'mgr.x'], ['osd.8', 'osd.9', 'osd.10', 'osd.11', 'mon.c']], 'seed': 6512, 'sha1': '1a6a3e742fd81c6d70be70552499bff767652c32', 'sleep_before_teardown': 0, 'subset': '28391/100000', 'suite': 'rados', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_d0bf63dd179c05751e4be49477389aa290c73089/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'd0bf63dd179c05751e4be49477389aa290c73089', 'targets': {'smithi059.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC78c5xWh8jIJwnkSb8HC90fPq7HoC3QQoqJO/0XfrGGer1Bajr3tHVTzQJNSKm5ca0Gz9ylZvP7ATr1ol+l1qc=', 'smithi081.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBEUm2ng7ce4gftKKX9BeQN3+wp1ILw30pdA4UEdIBM+YzbjV+UjG81RQlgZTiXAbNkYg+WqcgKMkc8IbJ8P9F8=', 'smithi179.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPTcNsLadPM7gOyBxe0Cie1W6T30xpnh0+lOPAZfC9BHHxJkMDg0ppgUuuau/+lqCPD1YVV5ugOmKDCdt7VKHns='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': '1a6a3e742fd81c6d70be70552499bff767652c32', 'cluster': 'ceph'}}, {'thrashosds': {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200}}, {'rados': {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_profile': {'crush-failure-domain': 'osd', 'k': 4, 'l': 3, 'm': 2, 'name': 'lrcprofile', 'plugin': 'lrc'}, 'objects': 50, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400, 'write_append_excl': False}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-18_21:00:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-19T02:01:55.822 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-19T02:01:55.851 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-19T02:01:55.854 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-19T02:01:55.860 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -d -m0777 -- /var/run/ceph 2024-04-19T02:01:55.899 DEBUG:teuthology.orchestra.run.smithi081:> sudo install -d -m0777 -- /var/run/ceph 2024-04-19T02:01:55.901 DEBUG:teuthology.orchestra.run.smithi179:> sudo install -d -m0777 -- /var/run/ceph 2024-04-19T02:01:55.914 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-19T02:01:55.914 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/scratch_devs of=/dev/stdout 2024-04-19T02:01:55.959 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-19T02:01:55.959 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_1 2024-04-19T02:01:56.007 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-19T02:01:56.007 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T02:01:56.007 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 877 Links: 1 2024-04-19T02:01:56.007 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T02:01:56.007 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-04-19 01:56:32.458236896 +0000 2024-04-19T02:01:56.007 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-04-19 01:56:32.206239147 +0000 2024-04-19T02:01:56.007 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-04-19 01:56:32.206239147 +0000 2024-04-19T02:01:56.007 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: - 2024-04-19T02:01:56.007 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-19T02:01:56.061 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-04-19T02:01:56.061 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-04-19T02:01:56.061 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000330039 s, 1.6 MB/s 2024-04-19T02:01:56.062 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-19T02:01:56.111 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_2 2024-04-19T02:01:56.158 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-19T02:01:56.158 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T02:01:56.158 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 891 Links: 1 2024-04-19T02:01:56.158 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T02:01:56.158 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-04-19 01:56:32.698234753 +0000 2024-04-19T02:01:56.158 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-04-19 01:56:32.694234788 +0000 2024-04-19T02:01:56.158 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-04-19 01:56:32.694234788 +0000 2024-04-19T02:01:56.158 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: - 2024-04-19T02:01:56.158 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-19T02:01:56.212 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-04-19T02:01:56.212 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-04-19T02:01:56.213 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.00044859 s, 1.1 MB/s 2024-04-19T02:01:56.214 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-19T02:01:56.263 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_3 2024-04-19T02:01:56.310 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-19T02:01:56.310 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T02:01:56.310 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 904 Links: 1 2024-04-19T02:01:56.310 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T02:01:56.310 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-04-19 01:56:33.442228107 +0000 2024-04-19T02:01:56.311 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-04-19 01:56:33.178230465 +0000 2024-04-19T02:01:56.311 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-04-19 01:56:33.178230465 +0000 2024-04-19T02:01:56.311 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: - 2024-04-19T02:01:56.311 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-19T02:01:56.364 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-04-19T02:01:56.364 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-04-19T02:01:56.364 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000407487 s, 1.3 MB/s 2024-04-19T02:01:56.366 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-19T02:01:56.415 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_4 2024-04-19T02:01:56.463 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-19T02:01:56.463 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T02:01:56.463 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 927 Links: 1 2024-04-19T02:01:56.463 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T02:01:56.463 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-04-19 01:56:33.954223534 +0000 2024-04-19T02:01:56.463 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-04-19 01:56:33.682225963 +0000 2024-04-19T02:01:56.463 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-04-19 01:56:33.682225963 +0000 2024-04-19T02:01:56.463 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: - 2024-04-19T02:01:56.463 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-19T02:01:56.517 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-04-19T02:01:56.517 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-04-19T02:01:56.517 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000349882 s, 1.5 MB/s 2024-04-19T02:01:56.519 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-19T02:01:56.568 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-19T02:01:56.568 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-19T02:01:56.568 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/scratch_devs of=/dev/stdout 2024-04-19T02:01:56.574 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-19T02:01:56.574 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_1 2024-04-19T02:01:56.623 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-19T02:01:56.623 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T02:01:56.623 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 879 Links: 1 2024-04-19T02:01:56.623 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T02:01:56.623 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-04-19 01:56:29.674135757 +0000 2024-04-19T02:01:56.623 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-04-19 01:56:29.430137936 +0000 2024-04-19T02:01:56.623 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-04-19 01:56:29.430137936 +0000 2024-04-19T02:01:56.623 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: - 2024-04-19T02:01:56.624 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-19T02:01:56.676 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-04-19T02:01:56.677 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-04-19T02:01:56.677 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000443457 s, 1.2 MB/s 2024-04-19T02:01:56.678 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-19T02:01:56.728 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_2 2024-04-19T02:01:56.774 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-19T02:01:56.774 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T02:01:56.774 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 893 Links: 1 2024-04-19T02:01:56.775 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T02:01:56.775 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-04-19 01:56:29.902133720 +0000 2024-04-19T02:01:56.775 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-04-19 01:56:29.898133756 +0000 2024-04-19T02:01:56.775 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-04-19 01:56:29.898133756 +0000 2024-04-19T02:01:56.775 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: - 2024-04-19T02:01:56.775 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-19T02:01:56.828 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-04-19T02:01:56.828 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-04-19T02:01:56.828 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000429 s, 1.2 MB/s 2024-04-19T02:01:56.830 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-19T02:01:56.880 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_3 2024-04-19T02:01:56.927 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-19T02:01:56.927 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T02:01:56.927 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 909 Links: 1 2024-04-19T02:01:56.927 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T02:01:56.927 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-04-19 01:56:30.662126931 +0000 2024-04-19T02:01:56.927 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-04-19 01:56:30.402129253 +0000 2024-04-19T02:01:56.927 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-04-19 01:56:30.402129253 +0000 2024-04-19T02:01:56.927 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: - 2024-04-19T02:01:56.927 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-19T02:01:56.980 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-04-19T02:01:56.981 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-04-19T02:01:56.981 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000388062 s, 1.3 MB/s 2024-04-19T02:01:56.982 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-19T02:01:57.032 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_4 2024-04-19T02:01:57.080 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-19T02:01:57.080 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T02:01:57.080 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 931 Links: 1 2024-04-19T02:01:57.080 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T02:01:57.080 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-04-19 01:56:31.194122180 +0000 2024-04-19T02:01:57.080 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-04-19 01:56:30.902124788 +0000 2024-04-19T02:01:57.081 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-04-19 01:56:30.902124788 +0000 2024-04-19T02:01:57.081 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: - 2024-04-19T02:01:57.081 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-19T02:01:57.132 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-04-19T02:01:57.132 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-04-19T02:01:57.132 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.00028189 s, 1.8 MB/s 2024-04-19T02:01:57.133 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-19T02:01:57.179 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-19T02:01:57.179 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-04-19T02:01:57.179 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/scratch_devs of=/dev/stdout 2024-04-19T02:01:57.185 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-19T02:01:57.185 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_1 2024-04-19T02:01:57.234 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-19T02:01:57.234 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T02:01:57.234 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 889 Links: 1 2024-04-19T02:01:57.234 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T02:01:57.235 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-04-19 01:56:37.318908035 +0000 2024-04-19T02:01:57.235 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-04-19 01:56:37.062910321 +0000 2024-04-19T02:01:57.235 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-04-19 01:56:37.062910321 +0000 2024-04-19T02:01:57.235 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: - 2024-04-19T02:01:57.235 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-19T02:01:57.288 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-04-19T02:01:57.288 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-04-19T02:01:57.288 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000390152 s, 1.3 MB/s 2024-04-19T02:01:57.290 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-19T02:01:57.339 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_2 2024-04-19T02:01:57.389 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-19T02:01:57.390 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T02:01:57.390 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 903 Links: 1 2024-04-19T02:01:57.390 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T02:01:57.390 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-04-19 01:56:37.790903819 +0000 2024-04-19T02:01:57.390 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-04-19 01:56:37.554905927 +0000 2024-04-19T02:01:57.390 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-04-19 01:56:37.554905927 +0000 2024-04-19T02:01:57.390 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: - 2024-04-19T02:01:57.390 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-19T02:01:57.443 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-04-19T02:01:57.443 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-04-19T02:01:57.443 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000422889 s, 1.2 MB/s 2024-04-19T02:01:57.445 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-19T02:01:57.495 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_3 2024-04-19T02:01:57.542 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-19T02:01:57.542 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T02:01:57.542 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 923 Links: 1 2024-04-19T02:01:57.542 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T02:01:57.542 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-04-19 01:56:38.294899318 +0000 2024-04-19T02:01:57.542 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-04-19 01:56:38.022901747 +0000 2024-04-19T02:01:57.542 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-04-19 01:56:38.022901747 +0000 2024-04-19T02:01:57.542 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: - 2024-04-19T02:01:57.543 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-19T02:01:57.596 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-04-19T02:01:57.596 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-04-19T02:01:57.596 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.00040298 s, 1.3 MB/s 2024-04-19T02:01:57.597 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-19T02:01:57.647 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_4 2024-04-19T02:01:57.694 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-19T02:01:57.694 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T02:01:57.694 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 938 Links: 1 2024-04-19T02:01:57.695 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T02:01:57.695 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-04-19 01:56:38.822894601 +0000 2024-04-19T02:01:57.695 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-04-19 01:56:38.538897137 +0000 2024-04-19T02:01:57.695 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-04-19 01:56:38.538897137 +0000 2024-04-19T02:01:57.695 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: - 2024-04-19T02:01:57.695 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-19T02:01:57.748 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-04-19T02:01:57.749 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-04-19T02:01:57.749 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000399161 s, 1.3 MB/s 2024-04-19T02:01:57.750 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-19T02:01:57.799 INFO:tasks.ceph:osd dev map: {'osd.8': '/dev/vg_nvme/lv_1', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.11': '/dev/vg_nvme/lv_4'} 2024-04-19T02:01:57.799 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi059.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi081.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi179.front.sepia.ceph.com'): {'osd.8': '/dev/vg_nvme/lv_1', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.11': '/dev/vg_nvme/lv_4'}} 2024-04-19T02:01:57.799 INFO:tasks.ceph:Generating config... 2024-04-19T02:01:57.800 INFO:tasks.ceph:[osd] debug monc = 20 2024-04-19T02:01:57.800 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2024-04-19T02:01:57.800 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2024-04-19T02:01:57.800 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-19T02:01:57.800 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-19T02:01:57.800 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-04-19T02:01:57.800 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-19T02:01:57.800 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-19T02:01:57.800 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-19T02:01:57.801 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-19T02:01:57.801 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-19T02:01:57.801 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-19T02:01:57.801 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-19T02:01:57.801 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-19T02:01:57.801 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-04-19T02:01:57.801 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2024-04-19T02:01:57.801 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-04-19T02:01:57.801 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-04-19T02:01:57.801 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-19T02:01:57.801 INFO:tasks.ceph:[osd] osd heartbeat use min delay socket = True 2024-04-19T02:01:57.801 INFO:tasks.ceph:[osd] osd max backfills = 6 2024-04-19T02:01:57.801 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-04-19T02:01:57.801 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-04-19T02:01:57.801 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-04-19T02:01:57.801 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-19T02:01:57.802 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-04-19T02:01:57.802 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-04-19T02:01:57.802 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-04-19T02:01:57.802 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-04-19T02:01:57.802 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-19T02:01:57.802 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-04-19T02:01:57.802 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-04-19T02:01:57.802 INFO:tasks.ceph:[global] osd_object_clean_region_max_num_intervals = 1000 2024-04-19T02:01:57.802 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-19T02:01:57.802 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-19T02:01:57.802 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-19T02:01:57.802 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-19T02:01:57.802 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-19T02:01:57.802 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-04-19T02:01:57.803 INFO:tasks.ceph:Setting up mon.a... 2024-04-19T02:01:57.803 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-19T02:01:57.838 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/ceph.keyring 2024-04-19T02:01:57.857 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-19T02:01:57.937 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-19T02:01:57.990 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.59'), ('mon.b', '172.21.15.81'), ('mon.c', '172.21.15.179')] 2024-04-19T02:01:57.990 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.59,172.21.15.81,172.21.15.179', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'osd_object_clean_region_max_num_intervals': 1000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub min interval': 60}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-04-19T02:01:57.991 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-19T02:01:57.991 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-19T02:01:58.039 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.59 --add b 172.21.15.81 --add c 172.21.15.179 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-19T02:01:58.126 INFO:teuthology.orchestra.run.smithi059.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-19T02:01:58.126 INFO:teuthology.orchestra.run.smithi059.stdout:monmaptool: generated fsid db9b8c36-b038-40a6-b749-a60ca7443e45 2024-04-19T02:01:58.126 INFO:teuthology.orchestra.run.smithi059.stdout:setting min_mon_release = quincy 2024-04-19T02:01:58.126 INFO:teuthology.orchestra.run.smithi059.stdout:epoch 0 2024-04-19T02:01:58.126 INFO:teuthology.orchestra.run.smithi059.stdout:fsid db9b8c36-b038-40a6-b749-a60ca7443e45 2024-04-19T02:01:58.126 INFO:teuthology.orchestra.run.smithi059.stdout:last_changed 2024-04-19T02:01:58.130773+0000 2024-04-19T02:01:58.126 INFO:teuthology.orchestra.run.smithi059.stdout:created 2024-04-19T02:01:58.130773+0000 2024-04-19T02:01:58.126 INFO:teuthology.orchestra.run.smithi059.stdout:min_mon_release 17 (quincy) 2024-04-19T02:01:58.127 INFO:teuthology.orchestra.run.smithi059.stdout:election_strategy: 1 2024-04-19T02:01:58.127 INFO:teuthology.orchestra.run.smithi059.stdout:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2024-04-19T02:01:58.127 INFO:teuthology.orchestra.run.smithi059.stdout:1: [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] mon.b 2024-04-19T02:01:58.127 INFO:teuthology.orchestra.run.smithi059.stdout:2: [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] mon.c 2024-04-19T02:01:58.127 INFO:teuthology.orchestra.run.smithi059.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-19T02:01:58.128 DEBUG:teuthology.orchestra.run.smithi059:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-19T02:01:58.133 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID db9b8c36-b038-40a6-b749-a60ca7443e45... 2024-04-19T02:01:58.134 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-19T02:01:58.179 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-19T02:01:58.182 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-19T02:01:58.206 INFO:teuthology.orchestra.run.smithi059.stdout:[global] 2024-04-19T02:01:58.206 INFO:teuthology.orchestra.run.smithi059.stdout: chdir = "" 2024-04-19T02:01:58.206 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-19T02:01:58.206 INFO:teuthology.orchestra.run.smithi059.stdout: auth supported = cephx 2024-04-19T02:01:58.206 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:01:58.206 INFO:teuthology.orchestra.run.smithi059.stdout: filestore xattr use omap = true 2024-04-19T02:01:58.206 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:01:58.206 INFO:teuthology.orchestra.run.smithi059.stdout: mon clock drift allowed = 1.000 2024-04-19T02:01:58.207 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:01:58.207 INFO:teuthology.orchestra.run.smithi059.stdout: osd crush chooseleaf type = 0 2024-04-19T02:01:58.207 INFO:teuthology.orchestra.run.smithi059.stdout: auth debug = true 2024-04-19T02:01:58.207 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:01:58.207 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on old message = true 2024-04-19T02:01:58.207 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on bug = true 2024-04-19T02:01:58.207 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:01:58.207 INFO:teuthology.orchestra.run.smithi059.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-19T02:01:58.207 INFO:teuthology.orchestra.run.smithi059.stdout: mon pg warn max object skew = 0 2024-04-19T02:01:58.207 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:01:58.207 INFO:teuthology.orchestra.run.smithi059.stdout: # disable pg_autoscaler by default for new pools 2024-04-19T02:01:58.207 INFO:teuthology.orchestra.run.smithi059.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-19T02:01:58.207 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:01:58.207 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default size = 2 2024-04-19T02:01:58.208 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:01:58.208 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow primary affinity = true 2024-04-19T02:01:58.208 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow pg remap = true 2024-04-19T02:01:58.208 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on legacy crush tunables = false 2024-04-19T02:01:58.208 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on crush straw calc version zero = false 2024-04-19T02:01:58.208 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on no sortbitwise = false 2024-04-19T02:01:58.208 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on osd down out interval zero = false 2024-04-19T02:01:58.208 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on too few osds = false 2024-04-19T02:01:58.208 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-19T02:01:58.208 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-19T02:01:58.208 INFO:teuthology.orchestra.run.smithi059.stdout: mon_allow_pool_size_one = true 2024-04-19T02:01:58.208 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:01:58.208 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-19T02:01:58.208 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:01:58.208 INFO:teuthology.orchestra.run.smithi059.stdout: osd default data pool replay window = 5 2024-04-19T02:01:58.209 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:01:58.209 INFO:teuthology.orchestra.run.smithi059.stdout: mon allow pool delete = true 2024-04-19T02:01:58.209 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:01:58.209 INFO:teuthology.orchestra.run.smithi059.stdout: mon cluster log file level = debug 2024-04-19T02:01:58.209 INFO:teuthology.orchestra.run.smithi059.stdout: debug asserts on shutdown = true 2024-04-19T02:01:58.209 INFO:teuthology.orchestra.run.smithi059.stdout: mon health detail to clog = false 2024-04-19T02:01:58.209 INFO:teuthology.orchestra.run.smithi059.stdout: mon host = "172.21.15.59,172.21.15.81,172.21.15.179" 2024-04-19T02:01:58.209 INFO:teuthology.orchestra.run.smithi059.stdout: mon client directed command retry = 5 2024-04-19T02:01:58.209 INFO:teuthology.orchestra.run.smithi059.stdout: mon election default strategy = 1 2024-04-19T02:01:58.209 INFO:teuthology.orchestra.run.smithi059.stdout: ms inject socket failures = 5000 2024-04-19T02:01:58.209 INFO:teuthology.orchestra.run.smithi059.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-04-19T02:01:58.209 INFO:teuthology.orchestra.run.smithi059.stdout: fsid = db9b8c36-b038-40a6-b749-a60ca7443e45 2024-04-19T02:01:58.209 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:01:58.209 INFO:teuthology.orchestra.run.smithi059.stdout:[osd] 2024-04-19T02:01:58.209 INFO:teuthology.orchestra.run.smithi059.stdout: osd journal size = 100 2024-04-19T02:01:58.209 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:01:58.209 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub load threshold = 5.0 2024-04-19T02:01:58.210 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub max interval = 120 2024-04-19T02:01:58.210 INFO:teuthology.orchestra.run.smithi059.stdout: osd mclock profile = high_recovery_ops 2024-04-19T02:01:58.210 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:01:58.210 INFO:teuthology.orchestra.run.smithi059.stdout: osd recover clone overlap = true 2024-04-19T02:01:58.210 INFO:teuthology.orchestra.run.smithi059.stdout: osd recovery max chunk = 1048576 2024-04-19T02:01:58.210 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:01:58.210 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug shutdown = true 2024-04-19T02:01:58.210 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug op order = true 2024-04-19T02:01:58.210 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug verify stray on activate = true 2024-04-19T02:01:58.210 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug trim objects = true 2024-04-19T02:01:58.211 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:01:58.211 INFO:teuthology.orchestra.run.smithi059.stdout: osd open classes on start = true 2024-04-19T02:01:58.211 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug pg log writeout = true 2024-04-19T02:01:58.211 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:01:58.211 INFO:teuthology.orchestra.run.smithi059.stdout: osd deep scrub update digest min age = 30 2024-04-19T02:01:58.211 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:01:58.211 INFO:teuthology.orchestra.run.smithi059.stdout: osd map max advance = 10 2024-04-19T02:01:58.211 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:01:58.211 INFO:teuthology.orchestra.run.smithi059.stdout: journal zero on create = true 2024-04-19T02:01:58.211 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:01:58.211 INFO:teuthology.orchestra.run.smithi059.stdout: filestore ondisk finisher threads = 3 2024-04-19T02:01:58.211 INFO:teuthology.orchestra.run.smithi059.stdout: filestore apply finisher threads = 3 2024-04-19T02:01:58.211 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:01:58.211 INFO:teuthology.orchestra.run.smithi059.stdout: bdev debug aio = true 2024-04-19T02:01:58.211 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug misdirected ops = true 2024-04-19T02:01:58.212 INFO:teuthology.orchestra.run.smithi059.stdout: debug monc = 20 2024-04-19T02:01:58.212 INFO:teuthology.orchestra.run.smithi059.stdout: bluefs allocator = hybrid 2024-04-19T02:01:58.212 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore allocator = hybrid 2024-04-19T02:01:58.212 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore block size = 96636764160 2024-04-19T02:01:58.212 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore fsck on mount = True 2024-04-19T02:01:58.212 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore zero block detection = True 2024-04-19T02:01:58.212 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluefs = 20 2024-04-19T02:01:58.212 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluestore = 20 2024-04-19T02:01:58.212 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-04-19T02:01:58.212 INFO:teuthology.orchestra.run.smithi059.stdout: debug osd = 20 2024-04-19T02:01:58.212 INFO:teuthology.orchestra.run.smithi059.stdout: debug rocksdb = 10 2024-04-19T02:01:58.212 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd backfillfull_ratio = 0.85 2024-04-19T02:01:58.212 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd full ratio = 0.9 2024-04-19T02:01:58.212 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd nearfull ratio = 0.8 2024-04-19T02:01:58.212 INFO:teuthology.orchestra.run.smithi059.stdout: osd blocked scrub grace period = 3600 2024-04-19T02:01:58.213 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug reject backfill probability = 0.3 2024-04-19T02:01:58.213 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug verify cached snaps = True 2024-04-19T02:01:58.213 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug verify missing on start = True 2024-04-19T02:01:58.213 INFO:teuthology.orchestra.run.smithi059.stdout: osd failsafe full ratio = 0.95 2024-04-19T02:01:58.213 INFO:teuthology.orchestra.run.smithi059.stdout: osd heartbeat use min delay socket = True 2024-04-19T02:01:58.213 INFO:teuthology.orchestra.run.smithi059.stdout: osd max backfills = 6 2024-04-19T02:01:58.213 INFO:teuthology.orchestra.run.smithi059.stdout: osd max markdown count = 1000 2024-04-19T02:01:58.213 INFO:teuthology.orchestra.run.smithi059.stdout: osd mclock override recovery settings = True 2024-04-19T02:01:58.213 INFO:teuthology.orchestra.run.smithi059.stdout: osd objectstore = bluestore 2024-04-19T02:01:58.213 INFO:teuthology.orchestra.run.smithi059.stdout: osd op queue = debug_random 2024-04-19T02:01:58.213 INFO:teuthology.orchestra.run.smithi059.stdout: osd op queue cut off = debug_random 2024-04-19T02:01:58.213 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub min interval = 60 2024-04-19T02:01:58.213 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:01:58.213 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr] 2024-04-19T02:01:58.213 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-04-19T02:01:58.213 INFO:teuthology.orchestra.run.smithi059.stdout: debug mgr = 20 2024-04-19T02:01:58.214 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-04-19T02:01:58.214 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-04-19T02:01:58.214 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-04-19T02:01:58.214 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-04-19T02:01:58.214 INFO:teuthology.orchestra.run.smithi059.stdout: mgr/telemetry/nag = false 2024-04-19T02:01:58.214 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:01:58.214 INFO:teuthology.orchestra.run.smithi059.stdout:[mon] 2024-04-19T02:01:58.214 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-04-19T02:01:58.214 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-04-19T02:01:58.214 INFO:teuthology.orchestra.run.smithi059.stdout: debug paxos = 20 2024-04-19T02:01:58.214 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-04-19T02:01:58.214 INFO:teuthology.orchestra.run.smithi059.stdout: mon data avail warn = 5 2024-04-19T02:01:58.214 INFO:teuthology.orchestra.run.smithi059.stdout: mon mgr mkfs grace = 240 2024-04-19T02:01:58.214 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-04-19T02:01:58.215 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd reporter subtree level = osd 2024-04-19T02:01:58.215 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd prime pg temp = true 2024-04-19T02:01:58.215 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-04-19T02:01:58.215 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:01:58.215 INFO:teuthology.orchestra.run.smithi059.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-19T02:01:58.215 INFO:teuthology.orchestra.run.smithi059.stdout: auth mon ticket ttl = 660 # 11m 2024-04-19T02:01:58.215 INFO:teuthology.orchestra.run.smithi059.stdout: auth service ticket ttl = 240 # 4m 2024-04-19T02:01:58.215 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:01:58.215 INFO:teuthology.orchestra.run.smithi059.stdout: # don't complain about insecure global_id in the test suite 2024-04-19T02:01:58.215 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-19T02:01:58.215 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-19T02:01:58.215 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:01:58.215 INFO:teuthology.orchestra.run.smithi059.stdout: # 1m isn't quite enough 2024-04-19T02:01:58.215 INFO:teuthology.orchestra.run.smithi059.stdout: mon_down_mkfs_grace = 2m 2024-04-19T02:01:58.216 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:01:58.216 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_filestore_osds = false 2024-04-19T02:01:58.216 INFO:teuthology.orchestra.run.smithi059.stdout: mon scrub interval = 300 2024-04-19T02:01:58.216 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:01:58.216 INFO:teuthology.orchestra.run.smithi059.stdout:[client] 2024-04-19T02:01:58.216 INFO:teuthology.orchestra.run.smithi059.stdout: rgw cache enabled = true 2024-04-19T02:01:58.216 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable ops log = true 2024-04-19T02:01:58.216 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable usage log = true 2024-04-19T02:01:58.216 INFO:teuthology.orchestra.run.smithi059.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-19T02:01:58.216 INFO:teuthology.orchestra.run.smithi059.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-19T02:01:58.216 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.a] 2024-04-19T02:01:58.216 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.b] 2024-04-19T02:01:58.216 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.c] 2024-04-19T02:01:58.217 INFO:teuthology.orchestra.run.smithi179.stdout:[global] 2024-04-19T02:01:58.217 INFO:teuthology.orchestra.run.smithi179.stdout: chdir = "" 2024-04-19T02:01:58.217 INFO:teuthology.orchestra.run.smithi179.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-19T02:01:58.217 INFO:teuthology.orchestra.run.smithi179.stdout: auth supported = cephx 2024-04-19T02:01:58.217 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-19T02:01:58.218 INFO:teuthology.orchestra.run.smithi179.stdout: filestore xattr use omap = true 2024-04-19T02:01:58.218 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-19T02:01:58.218 INFO:teuthology.orchestra.run.smithi179.stdout: mon clock drift allowed = 1.000 2024-04-19T02:01:58.218 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-19T02:01:58.218 INFO:teuthology.orchestra.run.smithi179.stdout: osd crush chooseleaf type = 0 2024-04-19T02:01:58.218 INFO:teuthology.orchestra.run.smithi179.stdout: auth debug = true 2024-04-19T02:01:58.218 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-19T02:01:58.218 INFO:teuthology.orchestra.run.smithi179.stdout: ms die on old message = true 2024-04-19T02:01:58.218 INFO:teuthology.orchestra.run.smithi179.stdout: ms die on bug = true 2024-04-19T02:01:58.218 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-19T02:01:58.218 INFO:teuthology.orchestra.run.smithi179.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-19T02:01:58.218 INFO:teuthology.orchestra.run.smithi179.stdout: mon pg warn max object skew = 0 2024-04-19T02:01:58.218 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-19T02:01:58.218 INFO:teuthology.orchestra.run.smithi179.stdout: # disable pg_autoscaler by default for new pools 2024-04-19T02:01:58.219 INFO:teuthology.orchestra.run.smithi179.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-19T02:01:58.219 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-19T02:01:58.219 INFO:teuthology.orchestra.run.smithi179.stdout: osd pool default size = 2 2024-04-19T02:01:58.219 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-19T02:01:58.219 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd allow primary affinity = true 2024-04-19T02:01:58.219 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd allow pg remap = true 2024-04-19T02:01:58.219 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on legacy crush tunables = false 2024-04-19T02:01:58.219 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on crush straw calc version zero = false 2024-04-19T02:01:58.219 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on no sortbitwise = false 2024-04-19T02:01:58.219 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on osd down out interval zero = false 2024-04-19T02:01:58.219 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on too few osds = false 2024-04-19T02:01:58.219 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-19T02:01:58.219 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-19T02:01:58.219 INFO:teuthology.orchestra.run.smithi179.stdout: mon_allow_pool_size_one = true 2024-04-19T02:01:58.219 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-19T02:01:58.219 INFO:teuthology.orchestra.run.smithi179.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-19T02:01:58.220 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-19T02:01:58.220 INFO:teuthology.orchestra.run.smithi179.stdout: osd default data pool replay window = 5 2024-04-19T02:01:58.220 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-19T02:01:58.220 INFO:teuthology.orchestra.run.smithi179.stdout: mon allow pool delete = true 2024-04-19T02:01:58.220 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-19T02:01:58.220 INFO:teuthology.orchestra.run.smithi179.stdout: mon cluster log file level = debug 2024-04-19T02:01:58.220 INFO:teuthology.orchestra.run.smithi179.stdout: debug asserts on shutdown = true 2024-04-19T02:01:58.220 INFO:teuthology.orchestra.run.smithi179.stdout: mon health detail to clog = false 2024-04-19T02:01:58.220 INFO:teuthology.orchestra.run.smithi179.stdout: mon host = "172.21.15.59,172.21.15.81,172.21.15.179" 2024-04-19T02:01:58.220 INFO:teuthology.orchestra.run.smithi179.stdout: mon client directed command retry = 5 2024-04-19T02:01:58.220 INFO:teuthology.orchestra.run.smithi179.stdout: mon election default strategy = 1 2024-04-19T02:01:58.220 INFO:teuthology.orchestra.run.smithi179.stdout: ms inject socket failures = 5000 2024-04-19T02:01:58.220 INFO:teuthology.orchestra.run.smithi179.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-04-19T02:01:58.220 INFO:teuthology.orchestra.run.smithi179.stdout: fsid = db9b8c36-b038-40a6-b749-a60ca7443e45 2024-04-19T02:01:58.220 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-19T02:01:58.221 INFO:teuthology.orchestra.run.smithi179.stdout:[osd] 2024-04-19T02:01:58.221 INFO:teuthology.orchestra.run.smithi179.stdout: osd journal size = 100 2024-04-19T02:01:58.221 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-19T02:01:58.221 INFO:teuthology.orchestra.run.smithi179.stdout: osd scrub load threshold = 5.0 2024-04-19T02:01:58.221 INFO:teuthology.orchestra.run.smithi179.stdout: osd scrub max interval = 120 2024-04-19T02:01:58.221 INFO:teuthology.orchestra.run.smithi179.stdout: osd mclock profile = high_recovery_ops 2024-04-19T02:01:58.221 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-19T02:01:58.221 INFO:teuthology.orchestra.run.smithi179.stdout: osd recover clone overlap = true 2024-04-19T02:01:58.221 INFO:teuthology.orchestra.run.smithi179.stdout: osd recovery max chunk = 1048576 2024-04-19T02:01:58.221 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-19T02:01:58.221 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug shutdown = true 2024-04-19T02:01:58.221 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug op order = true 2024-04-19T02:01:58.221 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug verify stray on activate = true 2024-04-19T02:01:58.221 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug trim objects = true 2024-04-19T02:01:58.221 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-19T02:01:58.221 INFO:teuthology.orchestra.run.smithi179.stdout: osd open classes on start = true 2024-04-19T02:01:58.221 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug pg log writeout = true 2024-04-19T02:01:58.222 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-19T02:01:58.222 INFO:teuthology.orchestra.run.smithi179.stdout: osd deep scrub update digest min age = 30 2024-04-19T02:01:58.222 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-19T02:01:58.222 INFO:teuthology.orchestra.run.smithi179.stdout: osd map max advance = 10 2024-04-19T02:01:58.222 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-19T02:01:58.222 INFO:teuthology.orchestra.run.smithi179.stdout: journal zero on create = true 2024-04-19T02:01:58.222 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-19T02:01:58.222 INFO:teuthology.orchestra.run.smithi179.stdout: filestore ondisk finisher threads = 3 2024-04-19T02:01:58.222 INFO:teuthology.orchestra.run.smithi179.stdout: filestore apply finisher threads = 3 2024-04-19T02:01:58.222 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-19T02:01:58.222 INFO:teuthology.orchestra.run.smithi179.stdout: bdev debug aio = true 2024-04-19T02:01:58.222 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug misdirected ops = true 2024-04-19T02:01:58.222 INFO:teuthology.orchestra.run.smithi179.stdout: debug monc = 20 2024-04-19T02:01:58.222 INFO:teuthology.orchestra.run.smithi179.stdout: bluefs allocator = hybrid 2024-04-19T02:01:58.223 INFO:teuthology.orchestra.run.smithi179.stdout: bluestore allocator = hybrid 2024-04-19T02:01:58.223 INFO:teuthology.orchestra.run.smithi179.stdout: bluestore block size = 96636764160 2024-04-19T02:01:58.223 INFO:teuthology.orchestra.run.smithi179.stdout: bluestore fsck on mount = True 2024-04-19T02:01:58.223 INFO:teuthology.orchestra.run.smithi179.stdout: bluestore zero block detection = True 2024-04-19T02:01:58.223 INFO:teuthology.orchestra.run.smithi179.stdout: debug bluefs = 20 2024-04-19T02:01:58.223 INFO:teuthology.orchestra.run.smithi179.stdout: debug bluestore = 20 2024-04-19T02:01:58.223 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2024-04-19T02:01:58.223 INFO:teuthology.orchestra.run.smithi179.stdout: debug osd = 20 2024-04-19T02:01:58.223 INFO:teuthology.orchestra.run.smithi179.stdout: debug rocksdb = 10 2024-04-19T02:01:58.223 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd backfillfull_ratio = 0.85 2024-04-19T02:01:58.223 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd full ratio = 0.9 2024-04-19T02:01:58.223 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd nearfull ratio = 0.8 2024-04-19T02:01:58.223 INFO:teuthology.orchestra.run.smithi179.stdout: osd blocked scrub grace period = 3600 2024-04-19T02:01:58.223 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug reject backfill probability = 0.3 2024-04-19T02:01:58.223 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug verify cached snaps = True 2024-04-19T02:01:58.223 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug verify missing on start = True 2024-04-19T02:01:58.224 INFO:teuthology.orchestra.run.smithi179.stdout: osd failsafe full ratio = 0.95 2024-04-19T02:01:58.224 INFO:teuthology.orchestra.run.smithi179.stdout: osd heartbeat use min delay socket = True 2024-04-19T02:01:58.224 INFO:teuthology.orchestra.run.smithi179.stdout: osd max backfills = 6 2024-04-19T02:01:58.224 INFO:teuthology.orchestra.run.smithi179.stdout: osd max markdown count = 1000 2024-04-19T02:01:58.224 INFO:teuthology.orchestra.run.smithi179.stdout: osd mclock override recovery settings = True 2024-04-19T02:01:58.224 INFO:teuthology.orchestra.run.smithi179.stdout: osd objectstore = bluestore 2024-04-19T02:01:58.224 INFO:teuthology.orchestra.run.smithi179.stdout: osd op queue = debug_random 2024-04-19T02:01:58.224 INFO:teuthology.orchestra.run.smithi179.stdout: osd op queue cut off = debug_random 2024-04-19T02:01:58.224 INFO:teuthology.orchestra.run.smithi179.stdout: osd scrub min interval = 60 2024-04-19T02:01:58.224 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-19T02:01:58.224 INFO:teuthology.orchestra.run.smithi179.stdout:[mgr] 2024-04-19T02:01:58.224 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2024-04-19T02:01:58.224 INFO:teuthology.orchestra.run.smithi179.stdout: debug mgr = 20 2024-04-19T02:01:58.224 INFO:teuthology.orchestra.run.smithi179.stdout: debug mon = 20 2024-04-19T02:01:58.225 INFO:teuthology.orchestra.run.smithi179.stdout: debug auth = 20 2024-04-19T02:01:58.225 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min pgs per osd = 4 2024-04-19T02:01:58.225 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min bytes per osd = 10 2024-04-19T02:01:58.225 INFO:teuthology.orchestra.run.smithi179.stdout: mgr/telemetry/nag = false 2024-04-19T02:01:58.225 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-19T02:01:58.225 INFO:teuthology.orchestra.run.smithi179.stdout:[mon] 2024-04-19T02:01:58.225 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2024-04-19T02:01:58.225 INFO:teuthology.orchestra.run.smithi179.stdout: debug mon = 20 2024-04-19T02:01:58.225 INFO:teuthology.orchestra.run.smithi179.stdout: debug paxos = 20 2024-04-19T02:01:58.225 INFO:teuthology.orchestra.run.smithi179.stdout: debug auth = 20 2024-04-19T02:01:58.225 INFO:teuthology.orchestra.run.smithi179.stdout: mon data avail warn = 5 2024-04-19T02:01:58.225 INFO:teuthology.orchestra.run.smithi179.stdout: mon mgr mkfs grace = 240 2024-04-19T02:01:58.225 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min pgs per osd = 4 2024-04-19T02:01:58.225 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd reporter subtree level = osd 2024-04-19T02:01:58.225 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd prime pg temp = true 2024-04-19T02:01:58.225 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min bytes per osd = 10 2024-04-19T02:01:58.225 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-19T02:01:58.226 INFO:teuthology.orchestra.run.smithi179.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-19T02:01:58.226 INFO:teuthology.orchestra.run.smithi179.stdout: auth mon ticket ttl = 660 # 11m 2024-04-19T02:01:58.226 INFO:teuthology.orchestra.run.smithi179.stdout: auth service ticket ttl = 240 # 4m 2024-04-19T02:01:58.226 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-19T02:01:58.226 INFO:teuthology.orchestra.run.smithi179.stdout: # don't complain about insecure global_id in the test suite 2024-04-19T02:01:58.226 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-19T02:01:58.226 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-19T02:01:58.226 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-19T02:01:58.226 INFO:teuthology.orchestra.run.smithi179.stdout: # 1m isn't quite enough 2024-04-19T02:01:58.226 INFO:teuthology.orchestra.run.smithi179.stdout: mon_down_mkfs_grace = 2m 2024-04-19T02:01:58.226 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-19T02:01:58.226 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_filestore_osds = false 2024-04-19T02:01:58.226 INFO:teuthology.orchestra.run.smithi179.stdout: mon scrub interval = 300 2024-04-19T02:01:58.226 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-19T02:01:58.226 INFO:teuthology.orchestra.run.smithi179.stdout:[client] 2024-04-19T02:01:58.227 INFO:teuthology.orchestra.run.smithi179.stdout: rgw cache enabled = true 2024-04-19T02:01:58.227 INFO:teuthology.orchestra.run.smithi179.stdout: rgw enable ops log = true 2024-04-19T02:01:58.227 INFO:teuthology.orchestra.run.smithi179.stdout: rgw enable usage log = true 2024-04-19T02:01:58.227 INFO:teuthology.orchestra.run.smithi179.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-19T02:01:58.227 INFO:teuthology.orchestra.run.smithi179.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-19T02:01:58.227 INFO:teuthology.orchestra.run.smithi179.stdout:[mon.a] 2024-04-19T02:01:58.227 INFO:teuthology.orchestra.run.smithi179.stdout:[mon.b] 2024-04-19T02:01:58.227 INFO:teuthology.orchestra.run.smithi179.stdout:[mon.c] 2024-04-19T02:01:58.228 INFO:teuthology.orchestra.run.smithi081.stdout:[global] 2024-04-19T02:01:58.228 INFO:teuthology.orchestra.run.smithi081.stdout: chdir = "" 2024-04-19T02:01:58.228 INFO:teuthology.orchestra.run.smithi081.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-19T02:01:58.228 INFO:teuthology.orchestra.run.smithi081.stdout: auth supported = cephx 2024-04-19T02:01:58.228 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-19T02:01:58.228 INFO:teuthology.orchestra.run.smithi081.stdout: filestore xattr use omap = true 2024-04-19T02:01:58.228 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-19T02:01:58.228 INFO:teuthology.orchestra.run.smithi081.stdout: mon clock drift allowed = 1.000 2024-04-19T02:01:58.229 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-19T02:01:58.229 INFO:teuthology.orchestra.run.smithi081.stdout: osd crush chooseleaf type = 0 2024-04-19T02:01:58.229 INFO:teuthology.orchestra.run.smithi081.stdout: auth debug = true 2024-04-19T02:01:58.229 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-19T02:01:58.229 INFO:teuthology.orchestra.run.smithi081.stdout: ms die on old message = true 2024-04-19T02:01:58.229 INFO:teuthology.orchestra.run.smithi081.stdout: ms die on bug = true 2024-04-19T02:01:58.229 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-19T02:01:58.229 INFO:teuthology.orchestra.run.smithi081.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-19T02:01:58.229 INFO:teuthology.orchestra.run.smithi081.stdout: mon pg warn max object skew = 0 2024-04-19T02:01:58.229 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-19T02:01:58.229 INFO:teuthology.orchestra.run.smithi081.stdout: # disable pg_autoscaler by default for new pools 2024-04-19T02:01:58.229 INFO:teuthology.orchestra.run.smithi081.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-19T02:01:58.229 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-19T02:01:58.229 INFO:teuthology.orchestra.run.smithi081.stdout: osd pool default size = 2 2024-04-19T02:01:58.229 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-19T02:01:58.229 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd allow primary affinity = true 2024-04-19T02:01:58.230 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd allow pg remap = true 2024-04-19T02:01:58.230 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on legacy crush tunables = false 2024-04-19T02:01:58.230 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on crush straw calc version zero = false 2024-04-19T02:01:58.230 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on no sortbitwise = false 2024-04-19T02:01:58.230 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on osd down out interval zero = false 2024-04-19T02:01:58.230 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on too few osds = false 2024-04-19T02:01:58.230 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-19T02:01:58.230 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-19T02:01:58.230 INFO:teuthology.orchestra.run.smithi081.stdout: mon_allow_pool_size_one = true 2024-04-19T02:01:58.230 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-19T02:01:58.230 INFO:teuthology.orchestra.run.smithi081.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-19T02:01:58.230 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-19T02:01:58.230 INFO:teuthology.orchestra.run.smithi081.stdout: osd default data pool replay window = 5 2024-04-19T02:01:58.230 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-19T02:01:58.231 INFO:teuthology.orchestra.run.smithi081.stdout: mon allow pool delete = true 2024-04-19T02:01:58.231 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-19T02:01:58.231 INFO:teuthology.orchestra.run.smithi081.stdout: mon cluster log file level = debug 2024-04-19T02:01:58.231 INFO:teuthology.orchestra.run.smithi081.stdout: debug asserts on shutdown = true 2024-04-19T02:01:58.231 INFO:teuthology.orchestra.run.smithi081.stdout: mon health detail to clog = false 2024-04-19T02:01:58.231 INFO:teuthology.orchestra.run.smithi081.stdout: mon host = "172.21.15.59,172.21.15.81,172.21.15.179" 2024-04-19T02:01:58.231 INFO:teuthology.orchestra.run.smithi081.stdout: mon client directed command retry = 5 2024-04-19T02:01:58.231 INFO:teuthology.orchestra.run.smithi081.stdout: mon election default strategy = 1 2024-04-19T02:01:58.231 INFO:teuthology.orchestra.run.smithi081.stdout: ms inject socket failures = 5000 2024-04-19T02:01:58.231 INFO:teuthology.orchestra.run.smithi081.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-04-19T02:01:58.231 INFO:teuthology.orchestra.run.smithi081.stdout: fsid = db9b8c36-b038-40a6-b749-a60ca7443e45 2024-04-19T02:01:58.231 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-19T02:01:58.231 INFO:teuthology.orchestra.run.smithi081.stdout:[osd] 2024-04-19T02:01:58.231 INFO:teuthology.orchestra.run.smithi081.stdout: osd journal size = 100 2024-04-19T02:01:58.231 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-19T02:01:58.231 INFO:teuthology.orchestra.run.smithi081.stdout: osd scrub load threshold = 5.0 2024-04-19T02:01:58.231 INFO:teuthology.orchestra.run.smithi081.stdout: osd scrub max interval = 120 2024-04-19T02:01:58.232 INFO:teuthology.orchestra.run.smithi081.stdout: osd mclock profile = high_recovery_ops 2024-04-19T02:01:58.232 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-19T02:01:58.232 INFO:teuthology.orchestra.run.smithi081.stdout: osd recover clone overlap = true 2024-04-19T02:01:58.232 INFO:teuthology.orchestra.run.smithi081.stdout: osd recovery max chunk = 1048576 2024-04-19T02:01:58.232 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-19T02:01:58.232 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug shutdown = true 2024-04-19T02:01:58.232 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug op order = true 2024-04-19T02:01:58.232 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug verify stray on activate = true 2024-04-19T02:01:58.232 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug trim objects = true 2024-04-19T02:01:58.232 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-19T02:01:58.232 INFO:teuthology.orchestra.run.smithi081.stdout: osd open classes on start = true 2024-04-19T02:01:58.232 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug pg log writeout = true 2024-04-19T02:01:58.232 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-19T02:01:58.233 INFO:teuthology.orchestra.run.smithi081.stdout: osd deep scrub update digest min age = 30 2024-04-19T02:01:58.233 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-19T02:01:58.233 INFO:teuthology.orchestra.run.smithi081.stdout: osd map max advance = 10 2024-04-19T02:01:58.233 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-19T02:01:58.233 INFO:teuthology.orchestra.run.smithi081.stdout: journal zero on create = true 2024-04-19T02:01:58.233 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-19T02:01:58.233 INFO:teuthology.orchestra.run.smithi081.stdout: filestore ondisk finisher threads = 3 2024-04-19T02:01:58.233 INFO:teuthology.orchestra.run.smithi081.stdout: filestore apply finisher threads = 3 2024-04-19T02:01:58.233 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-19T02:01:58.233 INFO:teuthology.orchestra.run.smithi081.stdout: bdev debug aio = true 2024-04-19T02:01:58.233 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug misdirected ops = true 2024-04-19T02:01:58.233 INFO:teuthology.orchestra.run.smithi081.stdout: debug monc = 20 2024-04-19T02:01:58.233 INFO:teuthology.orchestra.run.smithi081.stdout: bluefs allocator = hybrid 2024-04-19T02:01:58.233 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore allocator = hybrid 2024-04-19T02:01:58.233 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore block size = 96636764160 2024-04-19T02:01:58.233 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore fsck on mount = True 2024-04-19T02:01:58.234 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore zero block detection = True 2024-04-19T02:01:58.234 INFO:teuthology.orchestra.run.smithi081.stdout: debug bluefs = 20 2024-04-19T02:01:58.234 INFO:teuthology.orchestra.run.smithi081.stdout: debug bluestore = 20 2024-04-19T02:01:58.234 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-04-19T02:01:58.234 INFO:teuthology.orchestra.run.smithi081.stdout: debug osd = 20 2024-04-19T02:01:58.234 INFO:teuthology.orchestra.run.smithi081.stdout: debug rocksdb = 10 2024-04-19T02:01:58.234 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd backfillfull_ratio = 0.85 2024-04-19T02:01:58.234 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd full ratio = 0.9 2024-04-19T02:01:58.234 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd nearfull ratio = 0.8 2024-04-19T02:01:58.234 INFO:teuthology.orchestra.run.smithi081.stdout: osd blocked scrub grace period = 3600 2024-04-19T02:01:58.234 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug reject backfill probability = 0.3 2024-04-19T02:01:58.234 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug verify cached snaps = True 2024-04-19T02:01:58.234 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug verify missing on start = True 2024-04-19T02:01:58.234 INFO:teuthology.orchestra.run.smithi081.stdout: osd failsafe full ratio = 0.95 2024-04-19T02:01:58.234 INFO:teuthology.orchestra.run.smithi081.stdout: osd heartbeat use min delay socket = True 2024-04-19T02:01:58.235 INFO:teuthology.orchestra.run.smithi081.stdout: osd max backfills = 6 2024-04-19T02:01:58.235 INFO:teuthology.orchestra.run.smithi081.stdout: osd max markdown count = 1000 2024-04-19T02:01:58.235 INFO:teuthology.orchestra.run.smithi081.stdout: osd mclock override recovery settings = True 2024-04-19T02:01:58.235 INFO:teuthology.orchestra.run.smithi081.stdout: osd objectstore = bluestore 2024-04-19T02:01:58.235 INFO:teuthology.orchestra.run.smithi081.stdout: osd op queue = debug_random 2024-04-19T02:01:58.235 INFO:teuthology.orchestra.run.smithi081.stdout: osd op queue cut off = debug_random 2024-04-19T02:01:58.235 INFO:teuthology.orchestra.run.smithi081.stdout: osd scrub min interval = 60 2024-04-19T02:01:58.235 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-19T02:01:58.235 INFO:teuthology.orchestra.run.smithi081.stdout:[mgr] 2024-04-19T02:01:58.235 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-04-19T02:01:58.235 INFO:teuthology.orchestra.run.smithi081.stdout: debug mgr = 20 2024-04-19T02:01:58.235 INFO:teuthology.orchestra.run.smithi081.stdout: debug mon = 20 2024-04-19T02:01:58.235 INFO:teuthology.orchestra.run.smithi081.stdout: debug auth = 20 2024-04-19T02:01:58.235 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min pgs per osd = 4 2024-04-19T02:01:58.236 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min bytes per osd = 10 2024-04-19T02:01:58.236 INFO:teuthology.orchestra.run.smithi081.stdout: mgr/telemetry/nag = false 2024-04-19T02:01:58.236 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-19T02:01:58.236 INFO:teuthology.orchestra.run.smithi081.stdout:[mon] 2024-04-19T02:01:58.236 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-04-19T02:01:58.236 INFO:teuthology.orchestra.run.smithi081.stdout: debug mon = 20 2024-04-19T02:01:58.236 INFO:teuthology.orchestra.run.smithi081.stdout: debug paxos = 20 2024-04-19T02:01:58.236 INFO:teuthology.orchestra.run.smithi081.stdout: debug auth = 20 2024-04-19T02:01:58.236 INFO:teuthology.orchestra.run.smithi081.stdout: mon data avail warn = 5 2024-04-19T02:01:58.236 INFO:teuthology.orchestra.run.smithi081.stdout: mon mgr mkfs grace = 240 2024-04-19T02:01:58.236 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min pgs per osd = 4 2024-04-19T02:01:58.236 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd reporter subtree level = osd 2024-04-19T02:01:58.236 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd prime pg temp = true 2024-04-19T02:01:58.236 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min bytes per osd = 10 2024-04-19T02:01:58.236 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-19T02:01:58.236 INFO:teuthology.orchestra.run.smithi081.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-19T02:01:58.236 INFO:teuthology.orchestra.run.smithi081.stdout: auth mon ticket ttl = 660 # 11m 2024-04-19T02:01:58.237 INFO:teuthology.orchestra.run.smithi081.stdout: auth service ticket ttl = 240 # 4m 2024-04-19T02:01:58.237 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-19T02:01:58.237 INFO:teuthology.orchestra.run.smithi081.stdout: # don't complain about insecure global_id in the test suite 2024-04-19T02:01:58.237 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-19T02:01:58.237 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-19T02:01:58.237 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-19T02:01:58.237 INFO:teuthology.orchestra.run.smithi081.stdout: # 1m isn't quite enough 2024-04-19T02:01:58.237 INFO:teuthology.orchestra.run.smithi081.stdout: mon_down_mkfs_grace = 2m 2024-04-19T02:01:58.237 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-19T02:01:58.237 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_filestore_osds = false 2024-04-19T02:01:58.237 INFO:teuthology.orchestra.run.smithi081.stdout: mon scrub interval = 300 2024-04-19T02:01:58.237 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-19T02:01:58.237 INFO:teuthology.orchestra.run.smithi081.stdout:[client] 2024-04-19T02:01:58.237 INFO:teuthology.orchestra.run.smithi081.stdout: rgw cache enabled = true 2024-04-19T02:01:58.237 INFO:teuthology.orchestra.run.smithi081.stdout: rgw enable ops log = true 2024-04-19T02:01:58.237 INFO:teuthology.orchestra.run.smithi081.stdout: rgw enable usage log = true 2024-04-19T02:01:58.238 INFO:teuthology.orchestra.run.smithi081.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-19T02:01:58.238 INFO:teuthology.orchestra.run.smithi081.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-19T02:01:58.238 INFO:teuthology.orchestra.run.smithi081.stdout:[mon.a] 2024-04-19T02:01:58.238 INFO:teuthology.orchestra.run.smithi081.stdout:[mon.b] 2024-04-19T02:01:58.238 INFO:teuthology.orchestra.run.smithi081.stdout:[mon.c] 2024-04-19T02:01:58.239 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-19T02:01:58.239 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-19T02:01:58.321 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-19T02:01:58.321 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-19T02:01:58.321 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-19T02:01:58.327 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-19T02:01:58.327 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-19T02:01:58.373 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi059.front.sepia.ceph.com 2024-04-19T02:01:58.373 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-19T02:01:58.374 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-19T02:01:58.374 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-19T02:01:58.439 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-19T02:01:58.439 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-19T02:01:58.489 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi081.front.sepia.ceph.com 2024-04-19T02:01:58.489 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-19T02:01:58.489 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-19T02:01:58.489 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-19T02:01:58.511 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-19T02:01:58.512 DEBUG:teuthology.orchestra.run.smithi081:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-19T02:01:58.558 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi179.front.sepia.ceph.com 2024-04-19T02:01:58.558 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-04-19T02:01:58.558 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-19T02:01:58.558 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-19T02:01:58.581 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-04-19T02:01:58.581 DEBUG:teuthology.orchestra.run.smithi179:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-19T02:01:58.634 INFO:tasks.ceph:Setting up mon nodes... 2024-04-19T02:01:58.634 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-19T02:01:58.635 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-19T02:01:58.678 INFO:teuthology.orchestra.run.smithi081.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-19T02:01:58.687 INFO:tasks.ceph:Setting up mds nodes... 2024-04-19T02:01:58.687 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-19T02:01:58.687 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-19T02:01:58.725 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-19T02:01:58.738 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-19T02:01:58.738 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi059.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi081.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi179.front.sepia.ceph.com'): {'osd.8': '/dev/vg_nvme/lv_1', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.11': '/dev/vg_nvme/lv_4'}} 2024-04-19T02:01:58.738 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-19T02:01:58.794 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-19T02:01:58.794 INFO:tasks.ceph:role: osd.0 2024-04-19T02:01:58.794 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi059.front.sepia.ceph.com 2024-04-19T02:01:58.794 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-19T02:01:58.859 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T02:01:58.859 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T02:01:58.859 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T02:01:58.859 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T02:01:58.859 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T02:01:58.859 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-04-19T02:01:58.859 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T02:01:58.859 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T02:01:58.859 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T02:01:58.859 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T02:01:59.338 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-04-19T02:01:59.340 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-04-19T02:01:59.340 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-19T02:01:59.488 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-19T02:01:59.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T02:01:59.500 INFO:teuthology.orchestra.run.smithi059.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T02:01:59.500 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-19T02:01:59.558 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-19T02:01:59.558 INFO:tasks.ceph:role: osd.1 2024-04-19T02:01:59.558 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi059.front.sepia.ceph.com 2024-04-19T02:01:59.559 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-19T02:01:59.622 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T02:01:59.622 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T02:01:59.622 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T02:01:59.622 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T02:01:59.622 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T02:01:59.623 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-04-19T02:01:59.623 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T02:01:59.623 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T02:01:59.623 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T02:01:59.623 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T02:02:00.058 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-04-19T02:02:00.060 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-04-19T02:02:00.061 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-19T02:02:00.151 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-19T02:02:00.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T02:02:00.161 INFO:teuthology.orchestra.run.smithi059.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T02:02:00.161 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-19T02:02:00.216 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-19T02:02:00.216 INFO:tasks.ceph:role: osd.2 2024-04-19T02:02:00.216 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi059.front.sepia.ceph.com 2024-04-19T02:02:00.216 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-19T02:02:00.277 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T02:02:00.277 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T02:02:00.277 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T02:02:00.277 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T02:02:00.277 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T02:02:00.277 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-04-19T02:02:00.277 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T02:02:00.278 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T02:02:00.278 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T02:02:00.278 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T02:02:00.733 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-04-19T02:02:00.735 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-04-19T02:02:00.735 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-19T02:02:00.841 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-19T02:02:00.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T02:02:00.851 INFO:teuthology.orchestra.run.smithi059.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T02:02:00.851 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-19T02:02:00.903 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-19T02:02:00.903 INFO:tasks.ceph:role: osd.3 2024-04-19T02:02:00.903 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi059.front.sepia.ceph.com 2024-04-19T02:02:00.904 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-19T02:02:00.965 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T02:02:00.965 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T02:02:00.966 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T02:02:00.966 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T02:02:00.966 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T02:02:00.966 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-04-19T02:02:00.966 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T02:02:00.966 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T02:02:00.966 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T02:02:00.966 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T02:02:01.461 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-04-19T02:02:01.464 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-04-19T02:02:01.464 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-19T02:02:01.619 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-19T02:02:01.631 INFO:teuthology.orchestra.run.smithi059.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T02:02:01.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T02:02:01.632 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T02:02:01.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-19T02:02:01.744+0000 7f7fce1c57c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-19T02:02:01.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-19T02:02:01.748+0000 7f7fce1c57c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-19T02:02:01.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-19T02:02:01.748+0000 7f7fce1c57c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-19T02:02:01.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-19T02:02:01.748+0000 7f7fce1c57c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-19T02:02:01.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-19T02:02:01.748+0000 7f7fce1c57c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-19T02:02:01.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-19T02:02:01.756+0000 7f7fce1c57c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-19T02:02:04.735 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-19T02:02:04.749 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T02:02:04.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-19T02:02:04.828+0000 7f97084bc7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-19T02:02:04.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-19T02:02:04.832+0000 7f97084bc7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-19T02:02:04.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-19T02:02:04.832+0000 7f97084bc7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-19T02:02:04.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-19T02:02:04.832+0000 7f97084bc7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-19T02:02:04.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-19T02:02:04.832+0000 7f97084bc7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-19T02:02:04.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-19T02:02:04.832+0000 7f97084bc7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-19T02:02:07.814 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-19T02:02:07.871 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T02:02:07.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-19T02:02:07.952+0000 7f5916afa7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-19T02:02:07.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-19T02:02:07.952+0000 7f5916afa7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-19T02:02:07.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-19T02:02:07.952+0000 7f5916afa7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-19T02:02:07.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-19T02:02:07.952+0000 7f5916afa7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-19T02:02:07.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-19T02:02:07.952+0000 7f5916afa7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-19T02:02:07.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-19T02:02:07.956+0000 7f5916afa7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-19T02:02:10.938 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-19T02:02:10.995 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T02:02:11.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-19T02:02:11.080+0000 7f8f7f2fe7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-19T02:02:11.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-19T02:02:11.080+0000 7f8f7f2fe7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-19T02:02:11.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-19T02:02:11.080+0000 7f8f7f2fe7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-19T02:02:11.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-19T02:02:11.080+0000 7f8f7f2fe7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-19T02:02:11.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-19T02:02:11.080+0000 7f8f7f2fe7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-19T02:02:11.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-04-19T02:02:11.080+0000 7f8f7f2fe7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-19T02:02:14.074 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-19T02:02:14.130 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-19T02:02:14.145 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-19T02:02:14.145 INFO:tasks.ceph:role: osd.4 2024-04-19T02:02:14.145 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi081.front.sepia.ceph.com 2024-04-19T02:02:14.145 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-19T02:02:14.210 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T02:02:14.210 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T02:02:14.210 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T02:02:14.210 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T02:02:14.210 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T02:02:14.210 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2024-04-19T02:02:14.210 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T02:02:14.210 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T02:02:14.210 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T02:02:14.210 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T02:02:14.678 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2024-04-19T02:02:14.680 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2024-04-19T02:02:14.680 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-19T02:02:14.794 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-19T02:02:14.805 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T02:02:14.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T02:02:14.806 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-19T02:02:14.866 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-19T02:02:14.866 INFO:tasks.ceph:role: osd.5 2024-04-19T02:02:14.866 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi081.front.sepia.ceph.com 2024-04-19T02:02:14.866 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-19T02:02:14.929 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T02:02:14.929 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T02:02:14.929 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T02:02:14.929 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T02:02:14.929 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T02:02:14.929 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2024-04-19T02:02:14.930 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T02:02:14.930 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T02:02:14.930 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T02:02:14.930 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T02:02:15.360 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2024-04-19T02:02:15.362 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2024-04-19T02:02:15.363 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-19T02:02:15.430 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-19T02:02:15.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T02:02:15.483 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T02:02:15.484 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-19T02:02:15.538 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-19T02:02:15.538 INFO:tasks.ceph:role: osd.6 2024-04-19T02:02:15.538 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi081.front.sepia.ceph.com 2024-04-19T02:02:15.538 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-19T02:02:15.601 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T02:02:15.602 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T02:02:15.602 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T02:02:15.602 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T02:02:15.602 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T02:02:15.602 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2024-04-19T02:02:15.602 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T02:02:15.602 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T02:02:15.602 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T02:02:15.602 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T02:02:16.075 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2024-04-19T02:02:16.077 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2024-04-19T02:02:16.077 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-19T02:02:16.175 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-19T02:02:16.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T02:02:16.185 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T02:02:16.185 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-19T02:02:16.236 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-19T02:02:16.236 INFO:tasks.ceph:role: osd.7 2024-04-19T02:02:16.236 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi081.front.sepia.ceph.com 2024-04-19T02:02:16.236 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-19T02:02:16.299 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T02:02:16.299 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T02:02:16.299 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T02:02:16.299 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T02:02:16.299 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T02:02:16.299 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2024-04-19T02:02:16.299 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T02:02:16.299 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T02:02:16.299 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T02:02:16.299 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T02:02:16.763 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2024-04-19T02:02:16.765 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2024-04-19T02:02:16.765 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-19T02:02:16.870 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-19T02:02:16.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T02:02:16.879 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T02:02:16.879 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T02:02:17.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-19T02:02:17.000+0000 7fa14c7d97c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-19T02:02:17.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-19T02:02:17.004+0000 7fa14c7d97c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-19T02:02:17.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-19T02:02:17.004+0000 7fa14c7d97c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-19T02:02:17.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-19T02:02:17.004+0000 7fa14c7d97c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-19T02:02:17.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-19T02:02:17.004+0000 7fa14c7d97c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-19T02:02:17.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-19T02:02:17.012+0000 7fa14c7d97c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-19T02:02:19.982 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-19T02:02:19.996 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T02:02:20.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-19T02:02:20.076+0000 7efc6446c7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-19T02:02:20.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-19T02:02:20.076+0000 7efc6446c7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-19T02:02:20.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-19T02:02:20.076+0000 7efc6446c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-19T02:02:20.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-19T02:02:20.076+0000 7efc6446c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-19T02:02:20.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-19T02:02:20.076+0000 7efc6446c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-19T02:02:20.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-19T02:02:20.076+0000 7efc6446c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-19T02:02:23.048 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-19T02:02:23.103 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T02:02:23.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-19T02:02:23.184+0000 7fb5a89ca7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-19T02:02:23.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-19T02:02:23.184+0000 7fb5a89ca7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-19T02:02:23.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-19T02:02:23.184+0000 7fb5a89ca7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-19T02:02:23.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-19T02:02:23.184+0000 7fb5a89ca7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-19T02:02:23.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-19T02:02:23.184+0000 7fb5a89ca7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-19T02:02:23.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-19T02:02:23.184+0000 7fb5a89ca7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-19T02:02:26.157 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-19T02:02:26.219 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T02:02:26.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-19T02:02:26.300+0000 7f5aa3d967c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-19T02:02:26.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-19T02:02:26.304+0000 7f5aa3d967c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-19T02:02:26.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-19T02:02:26.304+0000 7f5aa3d967c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-19T02:02:26.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-19T02:02:26.304+0000 7f5aa3d967c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-19T02:02:26.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-19T02:02:26.304+0000 7f5aa3d967c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-19T02:02:26.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-19T02:02:26.304+0000 7f5aa3d967c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-19T02:02:29.282 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-19T02:02:29.339 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2024-04-19T02:02:29.353 INFO:tasks.ceph:roles_to_devs: {'osd.8': '/dev/vg_nvme/lv_1', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.11': '/dev/vg_nvme/lv_4'} 2024-04-19T02:02:29.353 INFO:tasks.ceph:role: osd.8 2024-04-19T02:02:29.353 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi179.front.sepia.ceph.com 2024-04-19T02:02:29.354 DEBUG:teuthology.orchestra.run.smithi179:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-19T02:02:29.417 INFO:teuthology.orchestra.run.smithi179.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T02:02:29.417 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T02:02:29.417 INFO:teuthology.orchestra.run.smithi179.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T02:02:29.417 INFO:teuthology.orchestra.run.smithi179.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T02:02:29.417 INFO:teuthology.orchestra.run.smithi179.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T02:02:29.417 INFO:teuthology.orchestra.run.smithi179.stdout: = sunit=0 swidth=0 blks 2024-04-19T02:02:29.417 INFO:teuthology.orchestra.run.smithi179.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T02:02:29.418 INFO:teuthology.orchestra.run.smithi179.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T02:02:29.418 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T02:02:29.418 INFO:teuthology.orchestra.run.smithi179.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T02:02:29.872 INFO:teuthology.orchestra.run.smithi179.stdout:Discarding blocks...Done. 2024-04-19T02:02:29.875 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi179.front.sepia.ceph.com -o noatime 2024-04-19T02:02:29.875 DEBUG:teuthology.orchestra.run.smithi179:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-8 2024-04-19T02:02:29.968 DEBUG:teuthology.orchestra.run.smithi179:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-8 2024-04-19T02:02:29.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T02:02:29.978 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T02:02:29.978 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2024-04-19T02:02:30.030 INFO:tasks.ceph:roles_to_devs: {'osd.8': '/dev/vg_nvme/lv_1', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.11': '/dev/vg_nvme/lv_4'} 2024-04-19T02:02:30.030 INFO:tasks.ceph:role: osd.9 2024-04-19T02:02:30.030 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi179.front.sepia.ceph.com 2024-04-19T02:02:30.030 DEBUG:teuthology.orchestra.run.smithi179:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-19T02:02:30.091 INFO:teuthology.orchestra.run.smithi179.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T02:02:30.091 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T02:02:30.091 INFO:teuthology.orchestra.run.smithi179.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T02:02:30.091 INFO:teuthology.orchestra.run.smithi179.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T02:02:30.091 INFO:teuthology.orchestra.run.smithi179.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T02:02:30.091 INFO:teuthology.orchestra.run.smithi179.stdout: = sunit=0 swidth=0 blks 2024-04-19T02:02:30.092 INFO:teuthology.orchestra.run.smithi179.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T02:02:30.092 INFO:teuthology.orchestra.run.smithi179.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T02:02:30.092 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T02:02:30.092 INFO:teuthology.orchestra.run.smithi179.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T02:02:30.558 INFO:teuthology.orchestra.run.smithi179.stdout:Discarding blocks...Done. 2024-04-19T02:02:30.560 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi179.front.sepia.ceph.com -o noatime 2024-04-19T02:02:30.560 DEBUG:teuthology.orchestra.run.smithi179:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-9 2024-04-19T02:02:30.685 DEBUG:teuthology.orchestra.run.smithi179:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-9 2024-04-19T02:02:30.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T02:02:30.695 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T02:02:30.695 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/osd/ceph-10 2024-04-19T02:02:30.752 INFO:tasks.ceph:roles_to_devs: {'osd.8': '/dev/vg_nvme/lv_1', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.11': '/dev/vg_nvme/lv_4'} 2024-04-19T02:02:30.752 INFO:tasks.ceph:role: osd.10 2024-04-19T02:02:30.752 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi179.front.sepia.ceph.com 2024-04-19T02:02:30.752 DEBUG:teuthology.orchestra.run.smithi179:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-19T02:02:30.813 INFO:teuthology.orchestra.run.smithi179.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T02:02:30.813 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T02:02:30.813 INFO:teuthology.orchestra.run.smithi179.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T02:02:30.813 INFO:teuthology.orchestra.run.smithi179.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T02:02:30.813 INFO:teuthology.orchestra.run.smithi179.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T02:02:30.813 INFO:teuthology.orchestra.run.smithi179.stdout: = sunit=0 swidth=0 blks 2024-04-19T02:02:30.813 INFO:teuthology.orchestra.run.smithi179.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T02:02:30.813 INFO:teuthology.orchestra.run.smithi179.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T02:02:30.813 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T02:02:30.813 INFO:teuthology.orchestra.run.smithi179.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T02:02:31.299 INFO:teuthology.orchestra.run.smithi179.stdout:Discarding blocks...Done. 2024-04-19T02:02:31.301 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi179.front.sepia.ceph.com -o noatime 2024-04-19T02:02:31.301 DEBUG:teuthology.orchestra.run.smithi179:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-10 2024-04-19T02:02:31.464 DEBUG:teuthology.orchestra.run.smithi179:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-10 2024-04-19T02:02:31.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T02:02:31.476 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T02:02:31.476 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/osd/ceph-11 2024-04-19T02:02:31.530 INFO:tasks.ceph:roles_to_devs: {'osd.8': '/dev/vg_nvme/lv_1', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.11': '/dev/vg_nvme/lv_4'} 2024-04-19T02:02:31.530 INFO:tasks.ceph:role: osd.11 2024-04-19T02:02:31.530 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi179.front.sepia.ceph.com 2024-04-19T02:02:31.530 DEBUG:teuthology.orchestra.run.smithi179:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-19T02:02:31.593 INFO:teuthology.orchestra.run.smithi179.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T02:02:31.593 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T02:02:31.593 INFO:teuthology.orchestra.run.smithi179.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T02:02:31.593 INFO:teuthology.orchestra.run.smithi179.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T02:02:31.593 INFO:teuthology.orchestra.run.smithi179.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T02:02:31.594 INFO:teuthology.orchestra.run.smithi179.stdout: = sunit=0 swidth=0 blks 2024-04-19T02:02:31.594 INFO:teuthology.orchestra.run.smithi179.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T02:02:31.594 INFO:teuthology.orchestra.run.smithi179.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T02:02:31.594 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T02:02:31.594 INFO:teuthology.orchestra.run.smithi179.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T02:02:32.009 INFO:teuthology.orchestra.run.smithi179.stdout:Discarding blocks...Done. 2024-04-19T02:02:32.012 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi179.front.sepia.ceph.com -o noatime 2024-04-19T02:02:32.012 DEBUG:teuthology.orchestra.run.smithi179:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-11 2024-04-19T02:02:32.112 DEBUG:teuthology.orchestra.run.smithi179:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-11 2024-04-19T02:02:32.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T02:02:32.122 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T02:02:32.122 DEBUG:teuthology.orchestra.run.smithi179:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T02:02:32.298 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-19T02:02:32.299+0000 7f551c9787c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2024-04-19T02:02:32.301 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-19T02:02:32.303+0000 7f551c9787c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2024-04-19T02:02:32.301 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-19T02:02:32.303+0000 7f551c9787c0 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-04-19T02:02:32.301 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-19T02:02:32.303+0000 7f551c9787c0 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-04-19T02:02:32.301 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-19T02:02:32.303+0000 7f551c9787c0 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-04-19T02:02:32.306 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-19T02:02:32.307+0000 7f551c9787c0 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2024-04-19T02:02:35.303 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-04-19T02:02:35.317 DEBUG:teuthology.orchestra.run.smithi179:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T02:02:35.398 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-19T02:02:35.399+0000 7f19100087c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2024-04-19T02:02:35.398 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-19T02:02:35.399+0000 7f19100087c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2024-04-19T02:02:35.399 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-19T02:02:35.399+0000 7f19100087c0 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-04-19T02:02:35.399 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-19T02:02:35.399+0000 7f19100087c0 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-04-19T02:02:35.399 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-19T02:02:35.399+0000 7f19100087c0 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-04-19T02:02:35.399 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-19T02:02:35.399+0000 7f19100087c0 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2024-04-19T02:02:38.361 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2024-04-19T02:02:38.418 DEBUG:teuthology.orchestra.run.smithi179:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 10 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T02:02:38.504 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-19T02:02:38.503+0000 7fa49e96f7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-10/keyring: can't open /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2024-04-19T02:02:38.505 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-19T02:02:38.507+0000 7fa49e96f7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-10/keyring 2024-04-19T02:02:38.505 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-19T02:02:38.507+0000 7fa49e96f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2024-04-19T02:02:38.505 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-19T02:02:38.507+0000 7fa49e96f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2024-04-19T02:02:38.505 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-19T02:02:38.507+0000 7fa49e96f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2024-04-19T02:02:38.506 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-19T02:02:38.507+0000 7fa49e96f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-10) _read_fsid unparsable uuid 2024-04-19T02:02:41.493 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-04-19T02:02:41.549 DEBUG:teuthology.orchestra.run.smithi179:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 11 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T02:02:41.634 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-19T02:02:41.631+0000 7f41a6f237c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-11/keyring: can't open /var/lib/ceph/osd/ceph-11/keyring: (2) No such file or directory 2024-04-19T02:02:41.634 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-19T02:02:41.631+0000 7f41a6f237c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-11/keyring 2024-04-19T02:02:41.634 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-19T02:02:41.631+0000 7f41a6f237c0 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2024-04-19T02:02:41.634 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-19T02:02:41.631+0000 7f41a6f237c0 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2024-04-19T02:02:41.635 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-19T02:02:41.631+0000 7f41a6f237c0 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2024-04-19T02:02:41.635 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-19T02:02:41.631+0000 7f41a6f237c0 -1 bluestore(/var/lib/ceph/osd/ceph-11) _read_fsid unparsable uuid 2024-04-19T02:02:44.628 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-11 2024-04-19T02:02:44.685 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-19T02:02:44.685 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-19T02:02:44.685 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-19T02:02:44.700 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-19T02:02:44.700 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-19T02:02:44.755 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-19T02:02:44.755 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-19T02:02:44.810 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-19T02:02:44.810 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-19T02:02:44.866 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-19T02:02:44.866 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-19T02:02:44.881 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-19T02:02:44.881 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-19T02:02:44.934 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-19T02:02:44.934 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-19T02:02:44.989 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-19T02:02:44.989 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-19T02:02:45.045 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-19T02:02:45.045 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-19T02:02:45.097 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-04-19T02:02:45.097 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2024-04-19T02:02:45.111 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-04-19T02:02:45.111 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2024-04-19T02:02:45.165 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-04-19T02:02:45.165 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-10/keyring of=/dev/stdout 2024-04-19T02:02:45.221 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-04-19T02:02:45.221 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-11/keyring of=/dev/stdout 2024-04-19T02:02:45.276 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-19T02:02:45.276 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-19T02:02:45.282 INFO:tasks.ceph:Adding keys to all mons... 2024-04-19T02:02:45.282 DEBUG:teuthology.orchestra.run.smithi059:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-19T02:02:45.327 DEBUG:teuthology.orchestra.run.smithi081:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-19T02:02:45.330 DEBUG:teuthology.orchestra.run.smithi179:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-19T02:02:45.337 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.0] 2024-04-19T02:02:45.337 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCZ0CFmZnaKLBAAqTsWuuSGcSCPdtadLOB75g== 2024-04-19T02:02:45.337 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.1] 2024-04-19T02:02:45.337 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCc0CFmagy7MRAAbeP7gcdADrvPF8aFfjmsjg== 2024-04-19T02:02:45.337 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.2] 2024-04-19T02:02:45.337 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCf0CFm8fcXORAAeEimJTB4jPBPxHuJyrUh5A== 2024-04-19T02:02:45.337 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.3] 2024-04-19T02:02:45.337 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCj0CFmvUDwBBAAepZ6E+13omFd+qj4Ka2VKw== 2024-04-19T02:02:45.337 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr.x] 2024-04-19T02:02:45.337 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCW0CFmkd2yKBAAPmWCfMhjcafePKOnGAV5OA== 2024-04-19T02:02:45.337 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.4] 2024-04-19T02:02:45.338 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCp0CFmYytQABAA+7O2Zyv6yhKOJSghsdS3ig== 2024-04-19T02:02:45.338 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.5] 2024-04-19T02:02:45.338 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCs0CFmS4vHBBAAEM8Iky/5okBg5gsMmcQUDw== 2024-04-19T02:02:45.338 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.6] 2024-04-19T02:02:45.338 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCv0CFmsokyCxAA1f+HkUusGlyQkwgjMHGx4A== 2024-04-19T02:02:45.338 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.7] 2024-04-19T02:02:45.338 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCy0CFm9/IzEhAA68Nn0vfdJobPnKralbcpjA== 2024-04-19T02:02:45.338 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.8] 2024-04-19T02:02:45.338 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQC40CFmNhEREhAAd6+RFNSvtur/oI0P3b0ieg== 2024-04-19T02:02:45.338 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.9] 2024-04-19T02:02:45.338 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQC70CFmFAgCGBAActYtLFyZhPU+XaNsL+becA== 2024-04-19T02:02:45.338 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.10] 2024-04-19T02:02:45.338 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQC+0CFmnq1ZHhAA5G/VTXpMAY9Xt9wVSdThJg== 2024-04-19T02:02:45.338 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.11] 2024-04-19T02:02:45.339 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQDB0CFmPzPsJRAALBWV9pb2d3Gxbv2Ak+qYBQ== 2024-04-19T02:02:45.339 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-04-19T02:02:45.339 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCW0CFmad+CKxAAXLNlCL3aRNAPV0HWnga5vA== 2024-04-19T02:02:45.340 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.0] 2024-04-19T02:02:45.340 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCZ0CFmZnaKLBAAqTsWuuSGcSCPdtadLOB75g== 2024-04-19T02:02:45.340 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.1] 2024-04-19T02:02:45.340 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCc0CFmagy7MRAAbeP7gcdADrvPF8aFfjmsjg== 2024-04-19T02:02:45.340 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.2] 2024-04-19T02:02:45.340 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCf0CFm8fcXORAAeEimJTB4jPBPxHuJyrUh5A== 2024-04-19T02:02:45.340 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.3] 2024-04-19T02:02:45.340 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCj0CFmvUDwBBAAepZ6E+13omFd+qj4Ka2VKw== 2024-04-19T02:02:45.340 INFO:teuthology.orchestra.run.smithi081.stdout:[mgr.x] 2024-04-19T02:02:45.340 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCW0CFmkd2yKBAAPmWCfMhjcafePKOnGAV5OA== 2024-04-19T02:02:45.340 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.4] 2024-04-19T02:02:45.340 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCp0CFmYytQABAA+7O2Zyv6yhKOJSghsdS3ig== 2024-04-19T02:02:45.341 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.5] 2024-04-19T02:02:45.341 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCs0CFmS4vHBBAAEM8Iky/5okBg5gsMmcQUDw== 2024-04-19T02:02:45.341 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.6] 2024-04-19T02:02:45.341 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCv0CFmsokyCxAA1f+HkUusGlyQkwgjMHGx4A== 2024-04-19T02:02:45.341 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.7] 2024-04-19T02:02:45.341 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCy0CFm9/IzEhAA68Nn0vfdJobPnKralbcpjA== 2024-04-19T02:02:45.341 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.8] 2024-04-19T02:02:45.341 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQC40CFmNhEREhAAd6+RFNSvtur/oI0P3b0ieg== 2024-04-19T02:02:45.341 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.9] 2024-04-19T02:02:45.341 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQC70CFmFAgCGBAActYtLFyZhPU+XaNsL+becA== 2024-04-19T02:02:45.341 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.10] 2024-04-19T02:02:45.341 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQC+0CFmnq1ZHhAA5G/VTXpMAY9Xt9wVSdThJg== 2024-04-19T02:02:45.341 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.11] 2024-04-19T02:02:45.341 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQDB0CFmPzPsJRAALBWV9pb2d3Gxbv2Ak+qYBQ== 2024-04-19T02:02:45.342 INFO:teuthology.orchestra.run.smithi081.stdout:[client.0] 2024-04-19T02:02:45.342 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCW0CFmad+CKxAAXLNlCL3aRNAPV0HWnga5vA== 2024-04-19T02:02:45.343 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.0] 2024-04-19T02:02:45.343 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQCZ0CFmZnaKLBAAqTsWuuSGcSCPdtadLOB75g== 2024-04-19T02:02:45.343 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.1] 2024-04-19T02:02:45.343 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQCc0CFmagy7MRAAbeP7gcdADrvPF8aFfjmsjg== 2024-04-19T02:02:45.343 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.2] 2024-04-19T02:02:45.343 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQCf0CFm8fcXORAAeEimJTB4jPBPxHuJyrUh5A== 2024-04-19T02:02:45.344 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.3] 2024-04-19T02:02:45.344 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQCj0CFmvUDwBBAAepZ6E+13omFd+qj4Ka2VKw== 2024-04-19T02:02:45.344 INFO:teuthology.orchestra.run.smithi179.stdout:[mgr.x] 2024-04-19T02:02:45.344 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQCW0CFmkd2yKBAAPmWCfMhjcafePKOnGAV5OA== 2024-04-19T02:02:45.344 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.4] 2024-04-19T02:02:45.344 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQCp0CFmYytQABAA+7O2Zyv6yhKOJSghsdS3ig== 2024-04-19T02:02:45.344 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.5] 2024-04-19T02:02:45.344 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQCs0CFmS4vHBBAAEM8Iky/5okBg5gsMmcQUDw== 2024-04-19T02:02:45.344 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.6] 2024-04-19T02:02:45.344 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQCv0CFmsokyCxAA1f+HkUusGlyQkwgjMHGx4A== 2024-04-19T02:02:45.344 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.7] 2024-04-19T02:02:45.344 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQCy0CFm9/IzEhAA68Nn0vfdJobPnKralbcpjA== 2024-04-19T02:02:45.344 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.8] 2024-04-19T02:02:45.344 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQC40CFmNhEREhAAd6+RFNSvtur/oI0P3b0ieg== 2024-04-19T02:02:45.344 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.9] 2024-04-19T02:02:45.345 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQC70CFmFAgCGBAActYtLFyZhPU+XaNsL+becA== 2024-04-19T02:02:45.345 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.10] 2024-04-19T02:02:45.345 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQC+0CFmnq1ZHhAA5G/VTXpMAY9Xt9wVSdThJg== 2024-04-19T02:02:45.345 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.11] 2024-04-19T02:02:45.345 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQDB0CFmPzPsJRAALBWV9pb2d3Gxbv2Ak+qYBQ== 2024-04-19T02:02:45.345 INFO:teuthology.orchestra.run.smithi179.stdout:[client.0] 2024-04-19T02:02:45.345 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQCW0CFmad+CKxAAXLNlCL3aRNAPV0HWnga5vA== 2024-04-19T02:02:45.345 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:45.383 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:45.387 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:45.432 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:45.434 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:45.436 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:45.512 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:45.519 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:45.522 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:45.598 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:45.603 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:45.606 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:45.681 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-19T02:02:45.687 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-19T02:02:45.689 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-19T02:02:45.768 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:45.771 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:45.775 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:45.856 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:45.859 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:45.862 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:45.940 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:45.943 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:45.947 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:46.028 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:46.030 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:46.033 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:46.112 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:46.115 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:46.119 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:46.196 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:46.199 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:46.203 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:46.277 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:46.284 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:46.287 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:46.360 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:46.367 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:46.369 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T02:02:46.448 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-19T02:02:46.451 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-19T02:02:46.453 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-19T02:02:46.539 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-19T02:02:46.539 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-19T02:02:46.553 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-19T02:02:46.647 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-19T02:02:46.657 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-19T02:02:46.671 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-19T02:02:46.762 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-19T02:02:46.772 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-19T02:02:46.786 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-19T02:02:46.879 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-19T02:02:46.890 DEBUG:teuthology.orchestra.run.smithi059:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-19T02:02:46.892 DEBUG:teuthology.orchestra.run.smithi081:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-19T02:02:46.895 DEBUG:teuthology.orchestra.run.smithi179:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-19T02:02:46.936 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-19T02:02:46.937 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-19T02:02:46.937 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-19T02:02:46.939 INFO:tasks.ceph.mon.a:Started 2024-04-19T02:02:46.940 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-19T02:02:46.940 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-19T02:02:46.943 INFO:tasks.ceph.mon.b:Started 2024-04-19T02:02:46.943 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-19T02:02:46.943 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-19T02:02:46.982 INFO:tasks.ceph.mon.c:Started 2024-04-19T02:02:46.982 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-19T02:02:46.982 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-19T02:02:46.983 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-19T02:02:46.985 INFO:tasks.ceph.mgr.x:Started 2024-04-19T02:02:46.985 INFO:tasks.ceph:Setting crush tunables to default 2024-04-19T02:02:46.985 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph osd crush tunables default 2024-04-19T02:02:47.470 INFO:teuthology.orchestra.run.smithi059.stderr:adjusted tunables profile to default 2024-04-19T02:02:47.489 INFO:tasks.ceph:check_enable_crimson: False 2024-04-19T02:02:47.490 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-19T02:02:47.490 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-19T02:02:47.490 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-19T02:02:47.503 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-19T02:02:47.503 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-19T02:02:47.556 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-19T02:02:47.556 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-19T02:02:47.610 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-04-19T02:02:47.610 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-19T02:02:47.666 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-19T02:02:47.666 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-19T02:02:47.679 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-19T02:02:47.680 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-19T02:02:47.685 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-04-19T02:02:47.683+0000 7fcc0137c180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-19T02:02:47.697 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-19T02:02:47.697 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-19T02:02:47.752 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-19T02:02:47.753 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-19T02:02:47.809 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-04-19T02:02:47.810 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2024-04-19T02:02:47.824 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-04-19T02:02:47.824 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2024-04-19T02:02:47.880 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-04-19T02:02:47.880 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-10/fsid of=/dev/stdout 2024-04-19T02:02:47.936 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-04-19T02:02:47.936 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-11/fsid of=/dev/stdout 2024-04-19T02:02:47.992 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new dff2e9d5-c79f-4cc8-b68c-d1d35cc3dfeb 0 2024-04-19T02:02:48.201 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-04-19T02:02:48.199+0000 7fcc0137c180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-19T02:02:48.405 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-04-19T02:02:48.403+0000 7fcc0137c180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-19T02:02:48.531 INFO:teuthology.orchestra.run.smithi179.stdout:0 2024-04-19T02:02:48.551 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 30cb7aa1-0bf8-4bc8-8bc5-f8b59870624e 1 2024-04-19T02:02:48.760 INFO:teuthology.orchestra.run.smithi179.stdout:1 2024-04-19T02:02:48.779 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 2e8fca1b-e008-4e58-865e-09a658f6edd9 2 2024-04-19T02:02:48.952 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-04-19T02:02:48.951+0000 7fcc0137c180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-19T02:02:49.006 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-04-19T02:02:49.003+0000 7fcc0137c180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-19T02:02:49.008 INFO:teuthology.orchestra.run.smithi179.stdout:2 2024-04-19T02:02:49.026 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 32289c9a-b7d9-4bcd-9f66-fa68f0c9796b 3 2024-04-19T02:02:49.058 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-04-19T02:02:49.059+0000 7fcc0137c180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-19T02:02:49.134 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-04-19T02:02:49.135+0000 7fcc0137c180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-19T02:02:49.186 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-04-19T02:02:49.187+0000 7fcc0137c180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-19T02:02:49.235 INFO:teuthology.orchestra.run.smithi179.stdout:3 2024-04-19T02:02:49.254 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 844daeea-860b-4853-a759-994531e5de3c 4 2024-04-19T02:02:49.262 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-04-19T02:02:49.263+0000 7fcc0137c180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-19T02:02:49.315 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-04-19T02:02:49.315+0000 7fcc0137c180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-19T02:02:49.463 INFO:teuthology.orchestra.run.smithi179.stdout:4 2024-04-19T02:02:49.465 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-04-19T02:02:49.463+0000 7fcc0137c180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-19T02:02:49.481 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new f43ad412-74bb-43a6-95e7-25ee06bde025 5 2024-04-19T02:02:49.526 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-04-19T02:02:49.527+0000 7fcc0137c180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-19T02:02:49.688 INFO:teuthology.orchestra.run.smithi179.stdout:5 2024-04-19T02:02:49.706 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 07ccbc36-1f5d-4595-8578-09362b5e212f 6 2024-04-19T02:02:49.918 INFO:teuthology.orchestra.run.smithi179.stdout:6 2024-04-19T02:02:49.936 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 6b2247e7-ebf8-4c0e-89ae-e15985f59aa1 7 2024-04-19T02:02:50.138 INFO:teuthology.orchestra.run.smithi179.stdout:7 2024-04-19T02:02:50.156 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 7142efd0-2144-4973-a01d-69a064b9b51f 8 2024-04-19T02:02:50.255 INFO:tasks.ceph.mgr.x.smithi081.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-19T02:02:50.255 INFO:tasks.ceph.mgr.x.smithi081.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-19T02:02:50.256 INFO:tasks.ceph.mgr.x.smithi081.stderr: from numpy import show_config as show_numpy_config 2024-04-19T02:02:50.308 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-04-19T02:02:50.307+0000 7fcc0137c180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-19T02:02:50.366 INFO:teuthology.orchestra.run.smithi179.stdout:8 2024-04-19T02:02:50.384 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 30a29baa-53cc-4c29-ab41-af14ead5c539 9 2024-04-19T02:02:50.470 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-04-19T02:02:50.471+0000 7fcc0137c180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-19T02:02:50.591 INFO:teuthology.orchestra.run.smithi179.stdout:9 2024-04-19T02:02:50.609 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new fae84343-8b7a-4bd3-aeef-823f2f6058e3 10 2024-04-19T02:02:50.624 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-04-19T02:02:50.623+0000 7fcc0137c180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-19T02:02:50.686 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-04-19T02:02:50.687+0000 7fcc0137c180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-19T02:02:50.812 INFO:teuthology.orchestra.run.smithi179.stdout:10 2024-04-19T02:02:50.819 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-04-19T02:02:50.819+0000 7fcc0137c180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-19T02:02:50.830 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 35858134-6864-4da1-88c2-4b48166febab 11 2024-04-19T02:02:51.039 INFO:teuthology.orchestra.run.smithi179.stdout:11 2024-04-19T02:02:51.057 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-19T02:02:51.057 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-19T02:02:51.060 INFO:tasks.ceph.osd.0:Started 2024-04-19T02:02:51.060 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-19T02:02:51.060 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-19T02:02:51.062 INFO:tasks.ceph.osd.1:Started 2024-04-19T02:02:51.063 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-19T02:02:51.063 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-19T02:02:51.065 INFO:tasks.ceph.osd.2:Started 2024-04-19T02:02:51.065 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-19T02:02:51.065 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-19T02:02:51.070 INFO:tasks.ceph.osd.3:Started 2024-04-19T02:02:51.070 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-19T02:02:51.070 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-19T02:02:51.073 INFO:tasks.ceph.osd.4:Started 2024-04-19T02:02:51.073 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-19T02:02:51.073 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-19T02:02:51.075 INFO:tasks.ceph.osd.5:Started 2024-04-19T02:02:51.075 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-19T02:02:51.075 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-19T02:02:51.078 INFO:tasks.ceph.osd.6:Started 2024-04-19T02:02:51.078 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-19T02:02:51.078 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-19T02:02:51.080 INFO:tasks.ceph.osd.7:Started 2024-04-19T02:02:51.080 INFO:tasks.ceph.osd.8:Restarting daemon 2024-04-19T02:02:51.080 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2024-04-19T02:02:51.083 INFO:tasks.ceph.osd.8:Started 2024-04-19T02:02:51.083 INFO:tasks.ceph.osd.9:Restarting daemon 2024-04-19T02:02:51.083 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2024-04-19T02:02:51.085 INFO:tasks.ceph.osd.9:Started 2024-04-19T02:02:51.085 INFO:tasks.ceph.osd.10:Restarting daemon 2024-04-19T02:02:51.085 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 10 2024-04-19T02:02:51.087 INFO:tasks.ceph.osd.10:Started 2024-04-19T02:02:51.088 INFO:tasks.ceph.osd.11:Restarting daemon 2024-04-19T02:02:51.088 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 11 2024-04-19T02:02:51.090 INFO:tasks.ceph.osd.11:Started 2024-04-19T02:02:51.090 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-19T02:02:51.294 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:02:51.295 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":14,"fsid":"db9b8c36-b038-40a6-b749-a60ca7443e45","created":"2024-04-19T02:02:47.352143+0000","modified":"2024-04-19T02:02:51.039271+0000","last_up_change":"0.000000","last_in_change":"2024-04-19T02:02:51.039271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dff2e9d5-c79f-4cc8-b68c-d1d35cc3dfeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"30cb7aa1-0bf8-4bc8-8bc5-f8b59870624e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2e8fca1b-e008-4e58-865e-09a658f6edd9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"32289c9a-b7d9-4bcd-9f66-fa68f0c9796b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"844daeea-860b-4853-a759-994531e5de3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f43ad412-74bb-43a6-95e7-25ee06bde025","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"07ccbc36-1f5d-4595-8578-09362b5e212f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6b2247e7-ebf8-4c0e-89ae-e15985f59aa1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"7142efd0-2144-4973-a01d-69a064b9b51f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"30a29baa-53cc-4c29-ab41-af14ead5c539","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"fae84343-8b7a-4bd3-aeef-823f2f6058e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"35858134-6864-4da1-88c2-4b48166febab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T02:02:51.312 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-19T02:02:51.312 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-19T02:02:51.488 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-04-19T02:02:51.487+0000 7fcc0137c180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-19T02:02:51.564 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-04-19T02:02:51.563+0000 7fcc0137c180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-19T02:02:51.686 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:02:51.687+0000 7feed50897c0 -1 Falling back to public interface 2024-04-19T02:02:51.687 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:02:51.687+0000 7f887dcff7c0 -1 Falling back to public interface 2024-04-19T02:02:51.695 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:02:51.696+0000 7f254df717c0 -1 Falling back to public interface 2024-04-19T02:02:51.698 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:02:51.699+0000 7f1f95f957c0 -1 Falling back to public interface 2024-04-19T02:02:51.698 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:02:51.699+0000 7f85ca95e7c0 -1 Falling back to public interface 2024-04-19T02:02:51.698 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:02:51.699+0000 7ff2420897c0 -1 Falling back to public interface 2024-04-19T02:02:51.701 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:02:51.703+0000 7f95517cb7c0 -1 Falling back to public interface 2024-04-19T02:02:51.705 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:02:51.707+0000 7f11c78957c0 -1 Falling back to public interface 2024-04-19T02:02:51.707 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:02:51.708+0000 7f9b3c6b97c0 -1 Falling back to public interface 2024-04-19T02:02:51.713 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:02:51.715+0000 7f2b6ad137c0 -1 Falling back to public interface 2024-04-19T02:02:51.722 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:02:51.724+0000 7f07db9977c0 -1 Falling back to public interface 2024-04-19T02:02:51.723 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:02:51.724+0000 7ff7b45797c0 -1 Falling back to public interface 2024-04-19T02:02:51.745 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-04-19T02:02:51.743+0000 7fcc0137c180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-19T02:02:51.818 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-04-19T02:02:51.819+0000 7fcc0137c180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-19T02:02:52.104 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-04-19T02:02:52.103+0000 7fcc0137c180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-19T02:02:52.165 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-04-19T02:02:52.163+0000 7fcc0137c180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-19T02:02:52.521 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-19T02:02:52.715 INFO:teuthology.misc.health.smithi059.stdout: 2024-04-19T02:02:52.715 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":14,"fsid":"db9b8c36-b038-40a6-b749-a60ca7443e45","created":"2024-04-19T02:02:47.352143+0000","modified":"2024-04-19T02:02:51.039271+0000","last_up_change":"0.000000","last_in_change":"2024-04-19T02:02:51.039271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dff2e9d5-c79f-4cc8-b68c-d1d35cc3dfeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"30cb7aa1-0bf8-4bc8-8bc5-f8b59870624e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2e8fca1b-e008-4e58-865e-09a658f6edd9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"32289c9a-b7d9-4bcd-9f66-fa68f0c9796b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"844daeea-860b-4853-a759-994531e5de3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f43ad412-74bb-43a6-95e7-25ee06bde025","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"07ccbc36-1f5d-4595-8578-09362b5e212f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6b2247e7-ebf8-4c0e-89ae-e15985f59aa1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"7142efd0-2144-4973-a01d-69a064b9b51f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"30a29baa-53cc-4c29-ab41-af14ead5c539","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"fae84343-8b7a-4bd3-aeef-823f2f6058e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"35858134-6864-4da1-88c2-4b48166febab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T02:02:52.731 DEBUG:teuthology.misc:0 of 12 OSDs are up 2024-04-19T02:02:53.208 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-04-19T02:02:53.207+0000 7fcba87c9640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-19T02:02:53.208 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-04-19T02:02:53.207+0000 7fcba87c9640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-19T02:02:55.225 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:02:55.227+0000 7ff2420897c0 -1 osd.9 0 log_to_monitors true 2024-04-19T02:02:55.330 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:02:55.331+0000 7f95517cb7c0 -1 osd.11 0 log_to_monitors true 2024-04-19T02:02:55.356 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:02:55.355+0000 7f85ca95e7c0 -1 osd.5 0 log_to_monitors true 2024-04-19T02:02:55.411 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:02:55.411+0000 7f254df717c0 -1 osd.1 0 log_to_monitors true 2024-04-19T02:02:55.433 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:02:55.435+0000 7f11c78957c0 -1 osd.10 0 log_to_monitors true 2024-04-19T02:02:55.487 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:02:55.487+0000 7f887dcff7c0 -1 osd.4 0 log_to_monitors true 2024-04-19T02:02:55.507 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:02:55.507+0000 7f9b3c6b97c0 -1 osd.0 0 log_to_monitors true 2024-04-19T02:02:55.528 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:02:55.531+0000 7f2b6ad137c0 -1 osd.8 0 log_to_monitors true 2024-04-19T02:02:55.572 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:02:55.571+0000 7feed50897c0 -1 osd.6 0 log_to_monitors true 2024-04-19T02:02:55.605 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:02:55.603+0000 7f07db9977c0 -1 osd.2 0 log_to_monitors true 2024-04-19T02:02:55.620 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:02:55.619+0000 7ff7b45797c0 -1 osd.3 0 log_to_monitors true 2024-04-19T02:02:55.674 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:02:55.675+0000 7f1f95f957c0 -1 osd.7 0 log_to_monitors true 2024-04-19T02:02:56.493 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:02:56.495+0000 7f11c383e640 -1 osd.10 0 waiting for initial osdmap 2024-04-19T02:02:56.508 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:02:56.507+0000 7f11bf67b640 -1 osd.10 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T02:02:57.375 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:02:57.375+0000 7f85c6907640 -1 osd.5 0 waiting for initial osdmap 2024-04-19T02:02:57.399 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:02:57.399+0000 7f85be73c640 -1 osd.5 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T02:02:57.399 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:02:57.399+0000 7f254a72c640 -1 osd.1 0 waiting for initial osdmap 2024-04-19T02:02:57.423 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:02:57.423+0000 7f2541d4f640 -1 osd.1 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T02:02:57.475 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:02:57.475+0000 7f2b66cbc640 -1 osd.8 0 waiting for initial osdmap 2024-04-19T02:02:57.478 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:02:57.479+0000 7f1f91f3e640 -1 osd.7 0 waiting for initial osdmap 2024-04-19T02:02:57.479 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:02:57.479+0000 7feed1032640 -1 osd.6 0 waiting for initial osdmap 2024-04-19T02:02:57.482 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:02:57.483+0000 7ff7b0522640 -1 osd.3 0 waiting for initial osdmap 2024-04-19T02:02:57.491 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:02:57.491+0000 7f8879ca8640 -1 osd.4 0 waiting for initial osdmap 2024-04-19T02:02:57.498 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:02:57.499+0000 7f2b62af9640 -1 osd.8 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T02:02:57.506 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:02:57.507+0000 7f8875ae5640 -1 osd.4 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T02:02:57.507 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:02:57.507+0000 7feecce6f640 -1 osd.6 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T02:02:57.510 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:02:57.511+0000 7f1f8dd7b640 -1 osd.7 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T02:02:57.518 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:02:57.515+0000 7ff7ac35f640 -1 osd.3 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T02:02:57.817 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:02:57.819+0000 7ff23e844640 -1 osd.9 0 waiting for initial osdmap 2024-04-19T02:02:57.824 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:02:57.823+0000 7f954d774640 -1 osd.11 0 waiting for initial osdmap 2024-04-19T02:02:57.849 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:02:57.851+0000 7ff235e67640 -1 osd.9 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T02:02:57.849 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:02:57.851+0000 7f95455a9640 -1 osd.11 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T02:02:58.893 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:02:58.891+0000 7f9b38e74640 -1 osd.0 0 waiting for initial osdmap 2024-04-19T02:02:58.898 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:02:58.899+0000 7f07d7940640 -1 osd.2 0 waiting for initial osdmap 2024-04-19T02:02:58.922 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:02:58.923+0000 7f9b30497640 -1 osd.0 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T02:02:58.923 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:02:58.923+0000 7f07cf775640 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T02:02:59.942 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-19T02:03:00.255 INFO:teuthology.misc.health.smithi059.stdout: 2024-04-19T02:03:00.255 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":19,"fsid":"db9b8c36-b038-40a6-b749-a60ca7443e45","created":"2024-04-19T02:02:47.352143+0000","modified":"2024-04-19T02:02:59.513773+0000","last_up_change":"2024-04-19T02:02:59.513773+0000","last_in_change":"2024-04-19T02:02:51.039271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-19T02:02:58.287078+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dff2e9d5-c79f-4cc8-b68c-d1d35cc3dfeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1145933874},{"type":"v1","addr":"172.21.15.59:6809","nonce":1145933874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1145933874},{"type":"v1","addr":"172.21.15.59:6811","nonce":1145933874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1145933874},{"type":"v1","addr":"172.21.15.59:6815","nonce":1145933874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1145933874},{"type":"v1","addr":"172.21.15.59:6813","nonce":1145933874}]},"public_addr":"172.21.15.59:6809/1145933874","cluster_addr":"172.21.15.59:6811/1145933874","heartbeat_back_addr":"172.21.15.59:6815/1145933874","heartbeat_front_addr":"172.21.15.59:6813/1145933874","state":["exists","up"]},{"osd":1,"uuid":"30cb7aa1-0bf8-4bc8-8bc5-f8b59870624e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":3807848039},{"type":"v1","addr":"172.21.15.59:6801","nonce":3807848039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":3807848039},{"type":"v1","addr":"172.21.15.59:6803","nonce":3807848039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":3807848039},{"type":"v1","addr":"172.21.15.59:6807","nonce":3807848039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":3807848039},{"type":"v1","addr":"172.21.15.59:6805","nonce":3807848039}]},"public_addr":"172.21.15.59:6801/3807848039","cluster_addr":"172.21.15.59:6803/3807848039","heartbeat_back_addr":"172.21.15.59:6807/3807848039","heartbeat_front_addr":"172.21.15.59:6805/3807848039","state":["exists","up"]},{"osd":2,"uuid":"2e8fca1b-e008-4e58-865e-09a658f6edd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":420007638},{"type":"v1","addr":"172.21.15.59:6817","nonce":420007638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":420007638},{"type":"v1","addr":"172.21.15.59:6822","nonce":420007638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6829","nonce":420007638},{"type":"v1","addr":"172.21.15.59:6831","nonce":420007638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":420007638},{"type":"v1","addr":"172.21.15.59:6827","nonce":420007638}]},"public_addr":"172.21.15.59:6817/420007638","cluster_addr":"172.21.15.59:6822/420007638","heartbeat_back_addr":"172.21.15.59:6831/420007638","heartbeat_front_addr":"172.21.15.59:6827/420007638","state":["exists","up"]},{"osd":3,"uuid":"32289c9a-b7d9-4bcd-9f66-fa68f0c9796b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1791539182},{"type":"v1","addr":"172.21.15.59:6819","nonce":1791539182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6821","nonce":1791539182},{"type":"v1","addr":"172.21.15.59:6823","nonce":1791539182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1791539182},{"type":"v1","addr":"172.21.15.59:6830","nonce":1791539182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6825","nonce":1791539182},{"type":"v1","addr":"172.21.15.59:6826","nonce":1791539182}]},"public_addr":"172.21.15.59:6819/1791539182","cluster_addr":"172.21.15.59:6823/1791539182","heartbeat_back_addr":"172.21.15.59:6830/1791539182","heartbeat_front_addr":"172.21.15.59:6826/1791539182","state":["exists","up"]},{"osd":4,"uuid":"844daeea-860b-4853-a759-994531e5de3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":2921507000},{"type":"v1","addr":"172.21.15.81:6808","nonce":2921507000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":2921507000},{"type":"v1","addr":"172.21.15.81:6811","nonce":2921507000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":2921507000},{"type":"v1","addr":"172.21.15.81:6815","nonce":2921507000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2921507000},{"type":"v1","addr":"172.21.15.81:6813","nonce":2921507000}]},"public_addr":"172.21.15.81:6808/2921507000","cluster_addr":"172.21.15.81:6811/2921507000","heartbeat_back_addr":"172.21.15.81:6815/2921507000","heartbeat_front_addr":"172.21.15.81:6813/2921507000","state":["exists","up"]},{"osd":5,"uuid":"f43ad412-74bb-43a6-95e7-25ee06bde025","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3925467179},{"type":"v1","addr":"172.21.15.81:6818","nonce":3925467179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":3925467179},{"type":"v1","addr":"172.21.15.81:6821","nonce":3925467179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":3925467179},{"type":"v1","addr":"172.21.15.81:6826","nonce":3925467179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":3925467179},{"type":"v1","addr":"172.21.15.81:6824","nonce":3925467179}]},"public_addr":"172.21.15.81:6818/3925467179","cluster_addr":"172.21.15.81:6821/3925467179","heartbeat_back_addr":"172.21.15.81:6826/3925467179","heartbeat_front_addr":"172.21.15.81:6824/3925467179","state":["exists","up"]},{"osd":6,"uuid":"07ccbc36-1f5d-4595-8578-09362b5e212f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3714790214},{"type":"v1","addr":"172.21.15.81:6801","nonce":3714790214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3714790214},{"type":"v1","addr":"172.21.15.81:6803","nonce":3714790214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6807","nonce":3714790214},{"type":"v1","addr":"172.21.15.81:6809","nonce":3714790214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3714790214},{"type":"v1","addr":"172.21.15.81:6805","nonce":3714790214}]},"public_addr":"172.21.15.81:6801/3714790214","cluster_addr":"172.21.15.81:6803/3714790214","heartbeat_back_addr":"172.21.15.81:6809/3714790214","heartbeat_front_addr":"172.21.15.81:6805/3714790214","state":["exists","up"]},{"osd":7,"uuid":"6b2247e7-ebf8-4c0e-89ae-e15985f59aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":2170932617},{"type":"v1","addr":"172.21.15.81:6820","nonce":2170932617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":2170932617},{"type":"v1","addr":"172.21.15.81:6827","nonce":2170932617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":2170932617},{"type":"v1","addr":"172.21.15.81:6831","nonce":2170932617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":2170932617},{"type":"v1","addr":"172.21.15.81:6829","nonce":2170932617}]},"public_addr":"172.21.15.81:6820/2170932617","cluster_addr":"172.21.15.81:6827/2170932617","heartbeat_back_addr":"172.21.15.81:6831/2170932617","heartbeat_front_addr":"172.21.15.81:6829/2170932617","state":["exists","up"]},{"osd":8,"uuid":"7142efd0-2144-4973-a01d-69a064b9b51f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2053612068},{"type":"v1","addr":"172.21.15.179:6825","nonce":2053612068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2053612068},{"type":"v1","addr":"172.21.15.179:6827","nonce":2053612068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2053612068},{"type":"v1","addr":"172.21.15.179:6831","nonce":2053612068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2053612068},{"type":"v1","addr":"172.21.15.179:6829","nonce":2053612068}]},"public_addr":"172.21.15.179:6825/2053612068","cluster_addr":"172.21.15.179:6827/2053612068","heartbeat_back_addr":"172.21.15.179:6831/2053612068","heartbeat_front_addr":"172.21.15.179:6829/2053612068","state":["exists","up"]},{"osd":9,"uuid":"30a29baa-53cc-4c29-ab41-af14ead5c539","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2641050728},{"type":"v1","addr":"172.21.15.179:6801","nonce":2641050728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2641050728},{"type":"v1","addr":"172.21.15.179:6803","nonce":2641050728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2641050728},{"type":"v1","addr":"172.21.15.179:6807","nonce":2641050728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2641050728},{"type":"v1","addr":"172.21.15.179:6805","nonce":2641050728}]},"public_addr":"172.21.15.179:6801/2641050728","cluster_addr":"172.21.15.179:6803/2641050728","heartbeat_back_addr":"172.21.15.179:6807/2641050728","heartbeat_front_addr":"172.21.15.179:6805/2641050728","state":["exists","up"]},{"osd":10,"uuid":"fae84343-8b7a-4bd3-aeef-823f2f6058e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3465727116},{"type":"v1","addr":"172.21.15.179:6817","nonce":3465727116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3465727116},{"type":"v1","addr":"172.21.15.179:6819","nonce":3465727116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3465727116},{"type":"v1","addr":"172.21.15.179:6823","nonce":3465727116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3465727116},{"type":"v1","addr":"172.21.15.179:6821","nonce":3465727116}]},"public_addr":"172.21.15.179:6817/3465727116","cluster_addr":"172.21.15.179:6819/3465727116","heartbeat_back_addr":"172.21.15.179:6823/3465727116","heartbeat_front_addr":"172.21.15.179:6821/3465727116","state":["exists","up"]},{"osd":11,"uuid":"35858134-6864-4da1-88c2-4b48166febab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":4047209718},{"type":"v1","addr":"172.21.15.179:6809","nonce":4047209718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":4047209718},{"type":"v1","addr":"172.21.15.179:6811","nonce":4047209718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4047209718},{"type":"v1","addr":"172.21.15.179:6815","nonce":4047209718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4047209718},{"type":"v1","addr":"172.21.15.179:6813","nonce":4047209718}]},"public_addr":"172.21.15.179:6809/4047209718","cluster_addr":"172.21.15.179:6811/4047209718","heartbeat_back_addr":"172.21.15.179:6815/4047209718","heartbeat_front_addr":"172.21.15.179:6813/4047209718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.434032+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.621023+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.404135+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.624821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.705799+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.547852+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.272469+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.401570+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.341513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T02:03:00.273 DEBUG:teuthology.misc:12 of 12 OSDs are up 2024-04-19T02:03:00.273 INFO:tasks.ceph:Creating RBD pool 2024-04-19T02:03:00.273 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-19T02:03:01.538 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'rbd' created 2024-04-19T02:03:01.563 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-19T02:03:02.547 INFO:teuthology.orchestra.run.smithi059.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-19T02:03:02.577 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-19T02:03:02.578 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-19T02:03:02.578 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-19T02:03:02.578 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-19T02:03:02.578 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-19T02:03:02.924 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:03:02.941 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3659401199},{"type":"v1","addr":"172.21.15.81:6833","nonce":3659401199}]},"active_addr":"172.21.15.81:6833/3659401199","active_change":"2024-04-19T02:02:52.192611+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":4125690725}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":2157977180}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":350038057}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":2254406394}]}]} 2024-04-19T02:03:02.943 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-19T02:03:02.944 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-19T02:03:02.944 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-19T02:03:03.241 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:03:03.241 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":22,"fsid":"db9b8c36-b038-40a6-b749-a60ca7443e45","created":"2024-04-19T02:02:47.352143+0000","modified":"2024-04-19T02:03:02.543473+0000","last_up_change":"2024-04-19T02:02:59.513773+0000","last_in_change":"2024-04-19T02:02:51.039271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-19T02:02:58.287078+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-19T02:03:00.613285+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.5199999809265137,"score_stable":4.5199999809265137,"optimal_score":0.82999998331069946,"raw_score_acting":3.75,"raw_score_stable":3.75,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dff2e9d5-c79f-4cc8-b68c-d1d35cc3dfeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1145933874},{"type":"v1","addr":"172.21.15.59:6809","nonce":1145933874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1145933874},{"type":"v1","addr":"172.21.15.59:6811","nonce":1145933874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1145933874},{"type":"v1","addr":"172.21.15.59:6815","nonce":1145933874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1145933874},{"type":"v1","addr":"172.21.15.59:6813","nonce":1145933874}]},"public_addr":"172.21.15.59:6809/1145933874","cluster_addr":"172.21.15.59:6811/1145933874","heartbeat_back_addr":"172.21.15.59:6815/1145933874","heartbeat_front_addr":"172.21.15.59:6813/1145933874","state":["exists","up"]},{"osd":1,"uuid":"30cb7aa1-0bf8-4bc8-8bc5-f8b59870624e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":3807848039},{"type":"v1","addr":"172.21.15.59:6801","nonce":3807848039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":3807848039},{"type":"v1","addr":"172.21.15.59:6803","nonce":3807848039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":3807848039},{"type":"v1","addr":"172.21.15.59:6807","nonce":3807848039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":3807848039},{"type":"v1","addr":"172.21.15.59:6805","nonce":3807848039}]},"public_addr":"172.21.15.59:6801/3807848039","cluster_addr":"172.21.15.59:6803/3807848039","heartbeat_back_addr":"172.21.15.59:6807/3807848039","heartbeat_front_addr":"172.21.15.59:6805/3807848039","state":["exists","up"]},{"osd":2,"uuid":"2e8fca1b-e008-4e58-865e-09a658f6edd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":420007638},{"type":"v1","addr":"172.21.15.59:6817","nonce":420007638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":420007638},{"type":"v1","addr":"172.21.15.59:6822","nonce":420007638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6829","nonce":420007638},{"type":"v1","addr":"172.21.15.59:6831","nonce":420007638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":420007638},{"type":"v1","addr":"172.21.15.59:6827","nonce":420007638}]},"public_addr":"172.21.15.59:6817/420007638","cluster_addr":"172.21.15.59:6822/420007638","heartbeat_back_addr":"172.21.15.59:6831/420007638","heartbeat_front_addr":"172.21.15.59:6827/420007638","state":["exists","up"]},{"osd":3,"uuid":"32289c9a-b7d9-4bcd-9f66-fa68f0c9796b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1791539182},{"type":"v1","addr":"172.21.15.59:6819","nonce":1791539182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6821","nonce":1791539182},{"type":"v1","addr":"172.21.15.59:6823","nonce":1791539182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1791539182},{"type":"v1","addr":"172.21.15.59:6830","nonce":1791539182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6825","nonce":1791539182},{"type":"v1","addr":"172.21.15.59:6826","nonce":1791539182}]},"public_addr":"172.21.15.59:6819/1791539182","cluster_addr":"172.21.15.59:6823/1791539182","heartbeat_back_addr":"172.21.15.59:6830/1791539182","heartbeat_front_addr":"172.21.15.59:6826/1791539182","state":["exists","up"]},{"osd":4,"uuid":"844daeea-860b-4853-a759-994531e5de3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":2921507000},{"type":"v1","addr":"172.21.15.81:6808","nonce":2921507000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":2921507000},{"type":"v1","addr":"172.21.15.81:6811","nonce":2921507000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":2921507000},{"type":"v1","addr":"172.21.15.81:6815","nonce":2921507000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2921507000},{"type":"v1","addr":"172.21.15.81:6813","nonce":2921507000}]},"public_addr":"172.21.15.81:6808/2921507000","cluster_addr":"172.21.15.81:6811/2921507000","heartbeat_back_addr":"172.21.15.81:6815/2921507000","heartbeat_front_addr":"172.21.15.81:6813/2921507000","state":["exists","up"]},{"osd":5,"uuid":"f43ad412-74bb-43a6-95e7-25ee06bde025","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3925467179},{"type":"v1","addr":"172.21.15.81:6818","nonce":3925467179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":3925467179},{"type":"v1","addr":"172.21.15.81:6821","nonce":3925467179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":3925467179},{"type":"v1","addr":"172.21.15.81:6826","nonce":3925467179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":3925467179},{"type":"v1","addr":"172.21.15.81:6824","nonce":3925467179}]},"public_addr":"172.21.15.81:6818/3925467179","cluster_addr":"172.21.15.81:6821/3925467179","heartbeat_back_addr":"172.21.15.81:6826/3925467179","heartbeat_front_addr":"172.21.15.81:6824/3925467179","state":["exists","up"]},{"osd":6,"uuid":"07ccbc36-1f5d-4595-8578-09362b5e212f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3714790214},{"type":"v1","addr":"172.21.15.81:6801","nonce":3714790214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3714790214},{"type":"v1","addr":"172.21.15.81:6803","nonce":3714790214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6807","nonce":3714790214},{"type":"v1","addr":"172.21.15.81:6809","nonce":3714790214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3714790214},{"type":"v1","addr":"172.21.15.81:6805","nonce":3714790214}]},"public_addr":"172.21.15.81:6801/3714790214","cluster_addr":"172.21.15.81:6803/3714790214","heartbeat_back_addr":"172.21.15.81:6809/3714790214","heartbeat_front_addr":"172.21.15.81:6805/3714790214","state":["exists","up"]},{"osd":7,"uuid":"6b2247e7-ebf8-4c0e-89ae-e15985f59aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":2170932617},{"type":"v1","addr":"172.21.15.81:6820","nonce":2170932617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":2170932617},{"type":"v1","addr":"172.21.15.81:6827","nonce":2170932617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":2170932617},{"type":"v1","addr":"172.21.15.81:6831","nonce":2170932617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":2170932617},{"type":"v1","addr":"172.21.15.81:6829","nonce":2170932617}]},"public_addr":"172.21.15.81:6820/2170932617","cluster_addr":"172.21.15.81:6827/2170932617","heartbeat_back_addr":"172.21.15.81:6831/2170932617","heartbeat_front_addr":"172.21.15.81:6829/2170932617","state":["exists","up"]},{"osd":8,"uuid":"7142efd0-2144-4973-a01d-69a064b9b51f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2053612068},{"type":"v1","addr":"172.21.15.179:6825","nonce":2053612068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2053612068},{"type":"v1","addr":"172.21.15.179:6827","nonce":2053612068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2053612068},{"type":"v1","addr":"172.21.15.179:6831","nonce":2053612068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2053612068},{"type":"v1","addr":"172.21.15.179:6829","nonce":2053612068}]},"public_addr":"172.21.15.179:6825/2053612068","cluster_addr":"172.21.15.179:6827/2053612068","heartbeat_back_addr":"172.21.15.179:6831/2053612068","heartbeat_front_addr":"172.21.15.179:6829/2053612068","state":["exists","up"]},{"osd":9,"uuid":"30a29baa-53cc-4c29-ab41-af14ead5c539","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2641050728},{"type":"v1","addr":"172.21.15.179:6801","nonce":2641050728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2641050728},{"type":"v1","addr":"172.21.15.179:6803","nonce":2641050728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2641050728},{"type":"v1","addr":"172.21.15.179:6807","nonce":2641050728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2641050728},{"type":"v1","addr":"172.21.15.179:6805","nonce":2641050728}]},"public_addr":"172.21.15.179:6801/2641050728","cluster_addr":"172.21.15.179:6803/2641050728","heartbeat_back_addr":"172.21.15.179:6807/2641050728","heartbeat_front_addr":"172.21.15.179:6805/2641050728","state":["exists","up"]},{"osd":10,"uuid":"fae84343-8b7a-4bd3-aeef-823f2f6058e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3465727116},{"type":"v1","addr":"172.21.15.179:6817","nonce":3465727116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3465727116},{"type":"v1","addr":"172.21.15.179:6819","nonce":3465727116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3465727116},{"type":"v1","addr":"172.21.15.179:6823","nonce":3465727116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3465727116},{"type":"v1","addr":"172.21.15.179:6821","nonce":3465727116}]},"public_addr":"172.21.15.179:6817/3465727116","cluster_addr":"172.21.15.179:6819/3465727116","heartbeat_back_addr":"172.21.15.179:6823/3465727116","heartbeat_front_addr":"172.21.15.179:6821/3465727116","state":["exists","up"]},{"osd":11,"uuid":"35858134-6864-4da1-88c2-4b48166febab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":4047209718},{"type":"v1","addr":"172.21.15.179:6809","nonce":4047209718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":4047209718},{"type":"v1","addr":"172.21.15.179:6811","nonce":4047209718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4047209718},{"type":"v1","addr":"172.21.15.179:6815","nonce":4047209718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4047209718},{"type":"v1","addr":"172.21.15.179:6813","nonce":4047209718}]},"public_addr":"172.21.15.179:6809/4047209718","cluster_addr":"172.21.15.179:6811/4047209718","heartbeat_back_addr":"172.21.15.179:6815/4047209718","heartbeat_front_addr":"172.21.15.179:6813/4047209718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.527995+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.434032+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.570606+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.621023+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.511399+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.404135+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.624821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.705799+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.547852+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.272469+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.401570+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.341513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T02:03:03.261 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-19T02:03:03.261 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-19T02:03:03.568 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:03:03.568 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"fsid":"db9b8c36-b038-40a6-b749-a60ca7443e45","created":"2024-04-19T02:02:47.352143+0000","modified":"2024-04-19T02:03:03.551342+0000","last_up_change":"2024-04-19T02:02:59.513773+0000","last_in_change":"2024-04-19T02:02:51.039271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-19T02:02:58.287078+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-19T02:03:00.613285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.5199999809265137,"score_stable":4.5199999809265137,"optimal_score":0.82999998331069946,"raw_score_acting":3.75,"raw_score_stable":3.75,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dff2e9d5-c79f-4cc8-b68c-d1d35cc3dfeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1145933874},{"type":"v1","addr":"172.21.15.59:6809","nonce":1145933874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1145933874},{"type":"v1","addr":"172.21.15.59:6811","nonce":1145933874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1145933874},{"type":"v1","addr":"172.21.15.59:6815","nonce":1145933874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1145933874},{"type":"v1","addr":"172.21.15.59:6813","nonce":1145933874}]},"public_addr":"172.21.15.59:6809/1145933874","cluster_addr":"172.21.15.59:6811/1145933874","heartbeat_back_addr":"172.21.15.59:6815/1145933874","heartbeat_front_addr":"172.21.15.59:6813/1145933874","state":["exists","up"]},{"osd":1,"uuid":"30cb7aa1-0bf8-4bc8-8bc5-f8b59870624e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":3807848039},{"type":"v1","addr":"172.21.15.59:6801","nonce":3807848039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":3807848039},{"type":"v1","addr":"172.21.15.59:6803","nonce":3807848039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":3807848039},{"type":"v1","addr":"172.21.15.59:6807","nonce":3807848039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":3807848039},{"type":"v1","addr":"172.21.15.59:6805","nonce":3807848039}]},"public_addr":"172.21.15.59:6801/3807848039","cluster_addr":"172.21.15.59:6803/3807848039","heartbeat_back_addr":"172.21.15.59:6807/3807848039","heartbeat_front_addr":"172.21.15.59:6805/3807848039","state":["exists","up"]},{"osd":2,"uuid":"2e8fca1b-e008-4e58-865e-09a658f6edd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":420007638},{"type":"v1","addr":"172.21.15.59:6817","nonce":420007638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":420007638},{"type":"v1","addr":"172.21.15.59:6822","nonce":420007638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6829","nonce":420007638},{"type":"v1","addr":"172.21.15.59:6831","nonce":420007638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":420007638},{"type":"v1","addr":"172.21.15.59:6827","nonce":420007638}]},"public_addr":"172.21.15.59:6817/420007638","cluster_addr":"172.21.15.59:6822/420007638","heartbeat_back_addr":"172.21.15.59:6831/420007638","heartbeat_front_addr":"172.21.15.59:6827/420007638","state":["exists","up"]},{"osd":3,"uuid":"32289c9a-b7d9-4bcd-9f66-fa68f0c9796b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1791539182},{"type":"v1","addr":"172.21.15.59:6819","nonce":1791539182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6821","nonce":1791539182},{"type":"v1","addr":"172.21.15.59:6823","nonce":1791539182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1791539182},{"type":"v1","addr":"172.21.15.59:6830","nonce":1791539182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6825","nonce":1791539182},{"type":"v1","addr":"172.21.15.59:6826","nonce":1791539182}]},"public_addr":"172.21.15.59:6819/1791539182","cluster_addr":"172.21.15.59:6823/1791539182","heartbeat_back_addr":"172.21.15.59:6830/1791539182","heartbeat_front_addr":"172.21.15.59:6826/1791539182","state":["exists","up"]},{"osd":4,"uuid":"844daeea-860b-4853-a759-994531e5de3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":2921507000},{"type":"v1","addr":"172.21.15.81:6808","nonce":2921507000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":2921507000},{"type":"v1","addr":"172.21.15.81:6811","nonce":2921507000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":2921507000},{"type":"v1","addr":"172.21.15.81:6815","nonce":2921507000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2921507000},{"type":"v1","addr":"172.21.15.81:6813","nonce":2921507000}]},"public_addr":"172.21.15.81:6808/2921507000","cluster_addr":"172.21.15.81:6811/2921507000","heartbeat_back_addr":"172.21.15.81:6815/2921507000","heartbeat_front_addr":"172.21.15.81:6813/2921507000","state":["exists","up"]},{"osd":5,"uuid":"f43ad412-74bb-43a6-95e7-25ee06bde025","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3925467179},{"type":"v1","addr":"172.21.15.81:6818","nonce":3925467179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":3925467179},{"type":"v1","addr":"172.21.15.81:6821","nonce":3925467179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":3925467179},{"type":"v1","addr":"172.21.15.81:6826","nonce":3925467179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":3925467179},{"type":"v1","addr":"172.21.15.81:6824","nonce":3925467179}]},"public_addr":"172.21.15.81:6818/3925467179","cluster_addr":"172.21.15.81:6821/3925467179","heartbeat_back_addr":"172.21.15.81:6826/3925467179","heartbeat_front_addr":"172.21.15.81:6824/3925467179","state":["exists","up"]},{"osd":6,"uuid":"07ccbc36-1f5d-4595-8578-09362b5e212f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3714790214},{"type":"v1","addr":"172.21.15.81:6801","nonce":3714790214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3714790214},{"type":"v1","addr":"172.21.15.81:6803","nonce":3714790214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6807","nonce":3714790214},{"type":"v1","addr":"172.21.15.81:6809","nonce":3714790214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3714790214},{"type":"v1","addr":"172.21.15.81:6805","nonce":3714790214}]},"public_addr":"172.21.15.81:6801/3714790214","cluster_addr":"172.21.15.81:6803/3714790214","heartbeat_back_addr":"172.21.15.81:6809/3714790214","heartbeat_front_addr":"172.21.15.81:6805/3714790214","state":["exists","up"]},{"osd":7,"uuid":"6b2247e7-ebf8-4c0e-89ae-e15985f59aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":2170932617},{"type":"v1","addr":"172.21.15.81:6820","nonce":2170932617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":2170932617},{"type":"v1","addr":"172.21.15.81:6827","nonce":2170932617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":2170932617},{"type":"v1","addr":"172.21.15.81:6831","nonce":2170932617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":2170932617},{"type":"v1","addr":"172.21.15.81:6829","nonce":2170932617}]},"public_addr":"172.21.15.81:6820/2170932617","cluster_addr":"172.21.15.81:6827/2170932617","heartbeat_back_addr":"172.21.15.81:6831/2170932617","heartbeat_front_addr":"172.21.15.81:6829/2170932617","state":["exists","up"]},{"osd":8,"uuid":"7142efd0-2144-4973-a01d-69a064b9b51f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2053612068},{"type":"v1","addr":"172.21.15.179:6825","nonce":2053612068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2053612068},{"type":"v1","addr":"172.21.15.179:6827","nonce":2053612068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2053612068},{"type":"v1","addr":"172.21.15.179:6831","nonce":2053612068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2053612068},{"type":"v1","addr":"172.21.15.179:6829","nonce":2053612068}]},"public_addr":"172.21.15.179:6825/2053612068","cluster_addr":"172.21.15.179:6827/2053612068","heartbeat_back_addr":"172.21.15.179:6831/2053612068","heartbeat_front_addr":"172.21.15.179:6829/2053612068","state":["exists","up"]},{"osd":9,"uuid":"30a29baa-53cc-4c29-ab41-af14ead5c539","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2641050728},{"type":"v1","addr":"172.21.15.179:6801","nonce":2641050728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2641050728},{"type":"v1","addr":"172.21.15.179:6803","nonce":2641050728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2641050728},{"type":"v1","addr":"172.21.15.179:6807","nonce":2641050728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2641050728},{"type":"v1","addr":"172.21.15.179:6805","nonce":2641050728}]},"public_addr":"172.21.15.179:6801/2641050728","cluster_addr":"172.21.15.179:6803/2641050728","heartbeat_back_addr":"172.21.15.179:6807/2641050728","heartbeat_front_addr":"172.21.15.179:6805/2641050728","state":["exists","up"]},{"osd":10,"uuid":"fae84343-8b7a-4bd3-aeef-823f2f6058e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3465727116},{"type":"v1","addr":"172.21.15.179:6817","nonce":3465727116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3465727116},{"type":"v1","addr":"172.21.15.179:6819","nonce":3465727116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3465727116},{"type":"v1","addr":"172.21.15.179:6823","nonce":3465727116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3465727116},{"type":"v1","addr":"172.21.15.179:6821","nonce":3465727116}]},"public_addr":"172.21.15.179:6817/3465727116","cluster_addr":"172.21.15.179:6819/3465727116","heartbeat_back_addr":"172.21.15.179:6823/3465727116","heartbeat_front_addr":"172.21.15.179:6821/3465727116","state":["exists","up"]},{"osd":11,"uuid":"35858134-6864-4da1-88c2-4b48166febab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":4047209718},{"type":"v1","addr":"172.21.15.179:6809","nonce":4047209718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":4047209718},{"type":"v1","addr":"172.21.15.179:6811","nonce":4047209718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4047209718},{"type":"v1","addr":"172.21.15.179:6815","nonce":4047209718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4047209718},{"type":"v1","addr":"172.21.15.179:6813","nonce":4047209718}]},"public_addr":"172.21.15.179:6809/4047209718","cluster_addr":"172.21.15.179:6811/4047209718","heartbeat_back_addr":"172.21.15.179:6815/4047209718","heartbeat_front_addr":"172.21.15.179:6813/4047209718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.527995+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.434032+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.570606+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.621023+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.511399+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.404135+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.624821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.705799+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.547852+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.272469+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.401570+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.341513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T02:03:03.587 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-19T02:03:03.587 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-19T02:03:03.588 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-19T02:03:03.588 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-19T02:03:03.589 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-19T02:03:03.589 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-19T02:03:03.589 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-19T02:03:03.590 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-19T02:03:03.590 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-04-19T02:03:03.591 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-04-19T02:03:03.591 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-04-19T02:03:03.591 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-04-19T02:03:03.922 INFO:teuthology.orchestra.run.smithi059.stdout:81604378628 2024-04-19T02:03:03.922 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-19T02:03:03.926 INFO:teuthology.orchestra.run.smithi059.stdout:73014444036 2024-04-19T02:03:03.926 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-19T02:03:03.947 INFO:teuthology.orchestra.run.smithi059.stdout:77309411332 2024-04-19T02:03:03.948 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-04-19T02:03:03.963 INFO:teuthology.orchestra.run.smithi059.stdout:77309411331 2024-04-19T02:03:03.964 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-19T02:03:03.964 INFO:teuthology.orchestra.run.smithi059.stdout:73014444036 2024-04-19T02:03:03.965 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-19T02:03:03.966 INFO:teuthology.orchestra.run.smithi059.stdout:73014444036 2024-04-19T02:03:03.966 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-04-19T02:03:03.978 INFO:teuthology.orchestra.run.smithi059.stdout:81604378627 2024-04-19T02:03:03.978 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-19T02:03:04.029 INFO:teuthology.orchestra.run.smithi059.stdout:77309411331 2024-04-19T02:03:04.030 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-04-19T02:03:04.035 INFO:teuthology.orchestra.run.smithi059.stdout:77309411330 2024-04-19T02:03:04.035 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-19T02:03:04.045 INFO:teuthology.orchestra.run.smithi059.stdout:77309411332 2024-04-19T02:03:04.046 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-19T02:03:04.047 INFO:teuthology.orchestra.run.smithi059.stdout:77309411331 2024-04-19T02:03:04.048 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-19T02:03:04.057 INFO:teuthology.orchestra.run.smithi059.stdout:77309411332 2024-04-19T02:03:04.058 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-04-19T02:03:04.602 INFO:teuthology.orchestra.run.smithi059.stdout:77309411332 2024-04-19T02:03:04.645 INFO:teuthology.orchestra.run.smithi059.stdout:77309411331 2024-04-19T02:03:04.646 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411332 got 77309411332 for osd.9 2024-04-19T02:03:04.646 DEBUG:teuthology.parallel:result is None 2024-04-19T02:03:04.666 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411331 got 77309411331 for osd.3 2024-04-19T02:03:04.666 DEBUG:teuthology.parallel:result is None 2024-04-19T02:03:04.685 INFO:teuthology.orchestra.run.smithi059.stdout:73014444036 2024-04-19T02:03:04.695 INFO:teuthology.orchestra.run.smithi059.stdout:81604378628 2024-04-19T02:03:04.721 INFO:teuthology.orchestra.run.smithi059.stdout:73014444036 2024-04-19T02:03:04.724 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444036 for osd.5 2024-04-19T02:03:04.724 DEBUG:teuthology.parallel:result is None 2024-04-19T02:03:04.731 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378628 got 81604378628 for osd.2 2024-04-19T02:03:04.731 DEBUG:teuthology.parallel:result is None 2024-04-19T02:03:04.743 INFO:teuthology.orchestra.run.smithi059.stdout:81604378627 2024-04-19T02:03:04.754 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444036 for osd.10 2024-04-19T02:03:04.754 DEBUG:teuthology.parallel:result is None 2024-04-19T02:03:04.765 INFO:teuthology.orchestra.run.smithi059.stdout:73014444036 2024-04-19T02:03:04.766 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378627 got 81604378627 for osd.0 2024-04-19T02:03:04.766 DEBUG:teuthology.parallel:result is None 2024-04-19T02:03:04.789 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444036 for osd.1 2024-04-19T02:03:04.789 INFO:teuthology.orchestra.run.smithi059.stdout:77309411331 2024-04-19T02:03:04.789 DEBUG:teuthology.parallel:result is None 2024-04-19T02:03:04.805 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411331 got 77309411331 for osd.4 2024-04-19T02:03:04.805 DEBUG:teuthology.parallel:result is None 2024-04-19T02:03:04.841 INFO:teuthology.orchestra.run.smithi059.stdout:77309411331 2024-04-19T02:03:04.843 INFO:teuthology.orchestra.run.smithi059.stdout:77309411332 2024-04-19T02:03:04.851 INFO:teuthology.orchestra.run.smithi059.stdout:77309411332 2024-04-19T02:03:04.863 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411331 got 77309411331 for osd.11 2024-04-19T02:03:04.864 DEBUG:teuthology.parallel:result is None 2024-04-19T02:03:04.864 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411332 got 77309411332 for osd.8 2024-04-19T02:03:04.865 INFO:teuthology.orchestra.run.smithi059.stdout:77309411330 2024-04-19T02:03:04.865 DEBUG:teuthology.parallel:result is None 2024-04-19T02:03:04.872 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411332 got 77309411332 for osd.7 2024-04-19T02:03:04.872 DEBUG:teuthology.parallel:result is None 2024-04-19T02:03:04.882 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411330 got 77309411330 for osd.6 2024-04-19T02:03:04.882 DEBUG:teuthology.parallel:result is None 2024-04-19T02:03:04.882 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-19T02:03:04.882 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T02:03:05.203 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:03:05.203 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-19T02:03:05.220 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-04-19T02:03:04.205230+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":328240,"kb_used_data":3500,"kb_used_omap":18,"kb_used_meta":324077,"kb_avail":1132133840,"statfs":{"total":1159641169920,"available":1159305052160,"internally_reserved":0,"allocated":3584000,"data_stored":2000796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":331855161},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.738538+0000","last_change":"2024-04-19T02:03:02.570839+0000","last_active":"2024-04-19T02:03:03.738538+0000","last_peered":"2024-04-19T02:03:03.738538+0000","last_clean":"2024-04-19T02:03:03.738538+0000","last_became_active":"2024-04-19T02:03:02.570116+0000","last_became_peered":"2024-04-19T02:03:02.570116+0000","last_unstale":"2024-04-19T02:03:03.738538+0000","last_undegraded":"2024-04-19T02:03:03.738538+0000","last_fullsized":"2024-04-19T02:03:03.738538+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:23.134425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.556984+0000","last_change":"2024-04-19T02:03:02.988302+0000","last_active":"2024-04-19T02:03:03.556984+0000","last_peered":"2024-04-19T02:03:03.556984+0000","last_clean":"2024-04-19T02:03:03.556984+0000","last_became_active":"2024-04-19T02:03:02.988073+0000","last_became_peered":"2024-04-19T02:03:02.988073+0000","last_unstale":"2024-04-19T02:03:03.556984+0000","last_undegraded":"2024-04-19T02:03:03.556984+0000","last_fullsized":"2024-04-19T02:03:03.556984+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:28.455271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.884666+0000","last_change":"2024-04-19T02:03:02.595929+0000","last_active":"2024-04-19T02:03:03.884666+0000","last_peered":"2024-04-19T02:03:03.884666+0000","last_clean":"2024-04-19T02:03:03.884666+0000","last_became_active":"2024-04-19T02:03:02.595232+0000","last_became_peered":"2024-04-19T02:03:02.595232+0000","last_unstale":"2024-04-19T02:03:03.884666+0000","last_undegraded":"2024-04-19T02:03:03.884666+0000","last_fullsized":"2024-04-19T02:03:03.884666+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:04.077370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.556937+0000","last_change":"2024-04-19T02:03:02.597582+0000","last_active":"2024-04-19T02:03:03.556937+0000","last_peered":"2024-04-19T02:03:03.556937+0000","last_clean":"2024-04-19T02:03:03.556937+0000","last_became_active":"2024-04-19T02:03:02.597118+0000","last_became_peered":"2024-04-19T02:03:02.597118+0000","last_unstale":"2024-04-19T02:03:03.556937+0000","last_undegraded":"2024-04-19T02:03:03.556937+0000","last_fullsized":"2024-04-19T02:03:03.556937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:13.683143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.738491+0000","last_change":"2024-04-19T02:03:02.560133+0000","last_active":"2024-04-19T02:03:03.738491+0000","last_peered":"2024-04-19T02:03:03.738491+0000","last_clean":"2024-04-19T02:03:03.738491+0000","last_became_active":"2024-04-19T02:03:02.559377+0000","last_became_peered":"2024-04-19T02:03:02.559377+0000","last_unstale":"2024-04-19T02:03:03.738491+0000","last_undegraded":"2024-04-19T02:03:03.738491+0000","last_fullsized":"2024-04-19T02:03:03.738491+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:30.728400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-19T02:03:02.687449+0000","last_change":"2024-04-19T02:03:02.686352+0000","last_active":"2024-04-19T02:03:02.687449+0000","last_peered":"2024-04-19T02:03:02.687449+0000","last_clean":"2024-04-19T02:03:02.687449+0000","last_became_active":"2024-04-19T02:03:02.685552+0000","last_became_peered":"2024-04-19T02:03:02.685552+0000","last_unstale":"2024-04-19T02:03:02.687449+0000","last_undegraded":"2024-04-19T02:03:02.687449+0000","last_fullsized":"2024-04-19T02:03:02.687449+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:08.251488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.738484+0000","last_change":"2024-04-19T02:03:02.561076+0000","last_active":"2024-04-19T02:03:03.738484+0000","last_peered":"2024-04-19T02:03:03.738484+0000","last_clean":"2024-04-19T02:03:03.738484+0000","last_became_active":"2024-04-19T02:03:02.560210+0000","last_became_peered":"2024-04-19T02:03:02.560210+0000","last_unstale":"2024-04-19T02:03:03.738484+0000","last_undegraded":"2024-04-19T02:03:03.738484+0000","last_fullsized":"2024-04-19T02:03:03.738484+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:16.900004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.557982+0000","last_change":"2024-04-19T02:03:03.026806+0000","last_active":"2024-04-19T02:03:03.557982+0000","last_peered":"2024-04-19T02:03:03.557982+0000","last_clean":"2024-04-19T02:03:03.557982+0000","last_became_active":"2024-04-19T02:03:03.026063+0000","last_became_peered":"2024-04-19T02:03:03.026063+0000","last_unstale":"2024-04-19T02:03:03.557982+0000","last_undegraded":"2024-04-19T02:03:03.557982+0000","last_fullsized":"2024-04-19T02:03:03.557982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:19.673016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":191,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.738375+0000","last_change":"2024-04-19T02:02:59.974166+0000","last_active":"2024-04-19T02:03:03.738375+0000","last_peered":"2024-04-19T02:03:03.738375+0000","last_clean":"2024-04-19T02:03:03.738375+0000","last_became_active":"2024-04-19T02:02:59.972005+0000","last_became_peered":"2024-04-19T02:02:59.972005+0000","last_unstale":"2024-04-19T02:03:03.738375+0000","last_undegraded":"2024-04-19T02:03:03.738375+0000","last_fullsized":"2024-04-19T02:03:03.738375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:02:58.486225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:02:58.486225+0000","last_clean_scrub_stamp":"2024-04-19T02:02:58.486225+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:20.275707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":19,"seq":81604378628,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":19,"seq":81604378627,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411331,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411332,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411332,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28196,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343644,"statfs":{"total":96636764160,"available":96607891456,"internally_reserved":0,"allocated":741376,"data_stored":606874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411332,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":18,"seq":77309411330,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411331,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":18,"seq":77309411331,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":85534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":17,"seq":73014444036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":17,"seq":73014444036,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":606874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":17,"seq":73014444036,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T02:03:05.221 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T02:03:05.500 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:03:05.500 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-19T02:03:05.517 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-04-19T02:03:04.205230+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":328240,"kb_used_data":3500,"kb_used_omap":18,"kb_used_meta":324077,"kb_avail":1132133840,"statfs":{"total":1159641169920,"available":1159305052160,"internally_reserved":0,"allocated":3584000,"data_stored":2000796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":331855161},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.738538+0000","last_change":"2024-04-19T02:03:02.570839+0000","last_active":"2024-04-19T02:03:03.738538+0000","last_peered":"2024-04-19T02:03:03.738538+0000","last_clean":"2024-04-19T02:03:03.738538+0000","last_became_active":"2024-04-19T02:03:02.570116+0000","last_became_peered":"2024-04-19T02:03:02.570116+0000","last_unstale":"2024-04-19T02:03:03.738538+0000","last_undegraded":"2024-04-19T02:03:03.738538+0000","last_fullsized":"2024-04-19T02:03:03.738538+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:23.134425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.556984+0000","last_change":"2024-04-19T02:03:02.988302+0000","last_active":"2024-04-19T02:03:03.556984+0000","last_peered":"2024-04-19T02:03:03.556984+0000","last_clean":"2024-04-19T02:03:03.556984+0000","last_became_active":"2024-04-19T02:03:02.988073+0000","last_became_peered":"2024-04-19T02:03:02.988073+0000","last_unstale":"2024-04-19T02:03:03.556984+0000","last_undegraded":"2024-04-19T02:03:03.556984+0000","last_fullsized":"2024-04-19T02:03:03.556984+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:28.455271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.884666+0000","last_change":"2024-04-19T02:03:02.595929+0000","last_active":"2024-04-19T02:03:03.884666+0000","last_peered":"2024-04-19T02:03:03.884666+0000","last_clean":"2024-04-19T02:03:03.884666+0000","last_became_active":"2024-04-19T02:03:02.595232+0000","last_became_peered":"2024-04-19T02:03:02.595232+0000","last_unstale":"2024-04-19T02:03:03.884666+0000","last_undegraded":"2024-04-19T02:03:03.884666+0000","last_fullsized":"2024-04-19T02:03:03.884666+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:04.077370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.556937+0000","last_change":"2024-04-19T02:03:02.597582+0000","last_active":"2024-04-19T02:03:03.556937+0000","last_peered":"2024-04-19T02:03:03.556937+0000","last_clean":"2024-04-19T02:03:03.556937+0000","last_became_active":"2024-04-19T02:03:02.597118+0000","last_became_peered":"2024-04-19T02:03:02.597118+0000","last_unstale":"2024-04-19T02:03:03.556937+0000","last_undegraded":"2024-04-19T02:03:03.556937+0000","last_fullsized":"2024-04-19T02:03:03.556937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:13.683143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.738491+0000","last_change":"2024-04-19T02:03:02.560133+0000","last_active":"2024-04-19T02:03:03.738491+0000","last_peered":"2024-04-19T02:03:03.738491+0000","last_clean":"2024-04-19T02:03:03.738491+0000","last_became_active":"2024-04-19T02:03:02.559377+0000","last_became_peered":"2024-04-19T02:03:02.559377+0000","last_unstale":"2024-04-19T02:03:03.738491+0000","last_undegraded":"2024-04-19T02:03:03.738491+0000","last_fullsized":"2024-04-19T02:03:03.738491+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:30.728400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-19T02:03:02.687449+0000","last_change":"2024-04-19T02:03:02.686352+0000","last_active":"2024-04-19T02:03:02.687449+0000","last_peered":"2024-04-19T02:03:02.687449+0000","last_clean":"2024-04-19T02:03:02.687449+0000","last_became_active":"2024-04-19T02:03:02.685552+0000","last_became_peered":"2024-04-19T02:03:02.685552+0000","last_unstale":"2024-04-19T02:03:02.687449+0000","last_undegraded":"2024-04-19T02:03:02.687449+0000","last_fullsized":"2024-04-19T02:03:02.687449+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:08.251488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.738484+0000","last_change":"2024-04-19T02:03:02.561076+0000","last_active":"2024-04-19T02:03:03.738484+0000","last_peered":"2024-04-19T02:03:03.738484+0000","last_clean":"2024-04-19T02:03:03.738484+0000","last_became_active":"2024-04-19T02:03:02.560210+0000","last_became_peered":"2024-04-19T02:03:02.560210+0000","last_unstale":"2024-04-19T02:03:03.738484+0000","last_undegraded":"2024-04-19T02:03:03.738484+0000","last_fullsized":"2024-04-19T02:03:03.738484+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:16.900004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.557982+0000","last_change":"2024-04-19T02:03:03.026806+0000","last_active":"2024-04-19T02:03:03.557982+0000","last_peered":"2024-04-19T02:03:03.557982+0000","last_clean":"2024-04-19T02:03:03.557982+0000","last_became_active":"2024-04-19T02:03:03.026063+0000","last_became_peered":"2024-04-19T02:03:03.026063+0000","last_unstale":"2024-04-19T02:03:03.557982+0000","last_undegraded":"2024-04-19T02:03:03.557982+0000","last_fullsized":"2024-04-19T02:03:03.557982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:19.673016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":191,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.738375+0000","last_change":"2024-04-19T02:02:59.974166+0000","last_active":"2024-04-19T02:03:03.738375+0000","last_peered":"2024-04-19T02:03:03.738375+0000","last_clean":"2024-04-19T02:03:03.738375+0000","last_became_active":"2024-04-19T02:02:59.972005+0000","last_became_peered":"2024-04-19T02:02:59.972005+0000","last_unstale":"2024-04-19T02:03:03.738375+0000","last_undegraded":"2024-04-19T02:03:03.738375+0000","last_fullsized":"2024-04-19T02:03:03.738375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:02:58.486225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:02:58.486225+0000","last_clean_scrub_stamp":"2024-04-19T02:02:58.486225+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:20.275707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":19,"seq":81604378628,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":19,"seq":81604378627,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411331,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411332,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411332,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28196,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343644,"statfs":{"total":96636764160,"available":96607891456,"internally_reserved":0,"allocated":741376,"data_stored":606874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411332,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":18,"seq":77309411330,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411331,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":18,"seq":77309411331,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":85534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":17,"seq":73014444036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":17,"seq":73014444036,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":606874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":17,"seq":73014444036,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T02:03:05.518 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-19T02:03:05.518 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-19T02:03:05.518 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-19T02:03:05.518 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-19T02:03:05.832 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:03:05.832 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-19T02:03:05.850 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-19T02:03:05.850 INFO:teuthology.run_tasks:Running task thrashosds... 2024-04-19T02:03:05.861 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-19T02:03:05.861 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': '1a6a3e742fd81c6d70be70552499bff767652c32'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1a6a3e742fd81c6d70be70552499bff767652c32'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'd0bf63dd179c05751e4be49477389aa290c73089'}} 2024-04-19T02:03:05.861 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-04-19T02:03:05.861 INFO:tasks.thrashosds:Beginning thrashosds... 2024-04-19T02:03:05.861 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-19T02:03:05.861 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T02:03:06.151 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:03:06.151 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-19T02:03:06.167 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-04-19T02:03:04.205230+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":328240,"kb_used_data":3500,"kb_used_omap":18,"kb_used_meta":324077,"kb_avail":1132133840,"statfs":{"total":1159641169920,"available":1159305052160,"internally_reserved":0,"allocated":3584000,"data_stored":2000796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":331855161},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.738538+0000","last_change":"2024-04-19T02:03:02.570839+0000","last_active":"2024-04-19T02:03:03.738538+0000","last_peered":"2024-04-19T02:03:03.738538+0000","last_clean":"2024-04-19T02:03:03.738538+0000","last_became_active":"2024-04-19T02:03:02.570116+0000","last_became_peered":"2024-04-19T02:03:02.570116+0000","last_unstale":"2024-04-19T02:03:03.738538+0000","last_undegraded":"2024-04-19T02:03:03.738538+0000","last_fullsized":"2024-04-19T02:03:03.738538+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:23.134425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.556984+0000","last_change":"2024-04-19T02:03:02.988302+0000","last_active":"2024-04-19T02:03:03.556984+0000","last_peered":"2024-04-19T02:03:03.556984+0000","last_clean":"2024-04-19T02:03:03.556984+0000","last_became_active":"2024-04-19T02:03:02.988073+0000","last_became_peered":"2024-04-19T02:03:02.988073+0000","last_unstale":"2024-04-19T02:03:03.556984+0000","last_undegraded":"2024-04-19T02:03:03.556984+0000","last_fullsized":"2024-04-19T02:03:03.556984+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:28.455271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.884666+0000","last_change":"2024-04-19T02:03:02.595929+0000","last_active":"2024-04-19T02:03:03.884666+0000","last_peered":"2024-04-19T02:03:03.884666+0000","last_clean":"2024-04-19T02:03:03.884666+0000","last_became_active":"2024-04-19T02:03:02.595232+0000","last_became_peered":"2024-04-19T02:03:02.595232+0000","last_unstale":"2024-04-19T02:03:03.884666+0000","last_undegraded":"2024-04-19T02:03:03.884666+0000","last_fullsized":"2024-04-19T02:03:03.884666+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:04.077370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.556937+0000","last_change":"2024-04-19T02:03:02.597582+0000","last_active":"2024-04-19T02:03:03.556937+0000","last_peered":"2024-04-19T02:03:03.556937+0000","last_clean":"2024-04-19T02:03:03.556937+0000","last_became_active":"2024-04-19T02:03:02.597118+0000","last_became_peered":"2024-04-19T02:03:02.597118+0000","last_unstale":"2024-04-19T02:03:03.556937+0000","last_undegraded":"2024-04-19T02:03:03.556937+0000","last_fullsized":"2024-04-19T02:03:03.556937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:13.683143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.738491+0000","last_change":"2024-04-19T02:03:02.560133+0000","last_active":"2024-04-19T02:03:03.738491+0000","last_peered":"2024-04-19T02:03:03.738491+0000","last_clean":"2024-04-19T02:03:03.738491+0000","last_became_active":"2024-04-19T02:03:02.559377+0000","last_became_peered":"2024-04-19T02:03:02.559377+0000","last_unstale":"2024-04-19T02:03:03.738491+0000","last_undegraded":"2024-04-19T02:03:03.738491+0000","last_fullsized":"2024-04-19T02:03:03.738491+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:30.728400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-19T02:03:02.687449+0000","last_change":"2024-04-19T02:03:02.686352+0000","last_active":"2024-04-19T02:03:02.687449+0000","last_peered":"2024-04-19T02:03:02.687449+0000","last_clean":"2024-04-19T02:03:02.687449+0000","last_became_active":"2024-04-19T02:03:02.685552+0000","last_became_peered":"2024-04-19T02:03:02.685552+0000","last_unstale":"2024-04-19T02:03:02.687449+0000","last_undegraded":"2024-04-19T02:03:02.687449+0000","last_fullsized":"2024-04-19T02:03:02.687449+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:08.251488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.738484+0000","last_change":"2024-04-19T02:03:02.561076+0000","last_active":"2024-04-19T02:03:03.738484+0000","last_peered":"2024-04-19T02:03:03.738484+0000","last_clean":"2024-04-19T02:03:03.738484+0000","last_became_active":"2024-04-19T02:03:02.560210+0000","last_became_peered":"2024-04-19T02:03:02.560210+0000","last_unstale":"2024-04-19T02:03:03.738484+0000","last_undegraded":"2024-04-19T02:03:03.738484+0000","last_fullsized":"2024-04-19T02:03:03.738484+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:16.900004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.557982+0000","last_change":"2024-04-19T02:03:03.026806+0000","last_active":"2024-04-19T02:03:03.557982+0000","last_peered":"2024-04-19T02:03:03.557982+0000","last_clean":"2024-04-19T02:03:03.557982+0000","last_became_active":"2024-04-19T02:03:03.026063+0000","last_became_peered":"2024-04-19T02:03:03.026063+0000","last_unstale":"2024-04-19T02:03:03.557982+0000","last_undegraded":"2024-04-19T02:03:03.557982+0000","last_fullsized":"2024-04-19T02:03:03.557982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:19.673016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":191,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.738375+0000","last_change":"2024-04-19T02:02:59.974166+0000","last_active":"2024-04-19T02:03:03.738375+0000","last_peered":"2024-04-19T02:03:03.738375+0000","last_clean":"2024-04-19T02:03:03.738375+0000","last_became_active":"2024-04-19T02:02:59.972005+0000","last_became_peered":"2024-04-19T02:02:59.972005+0000","last_unstale":"2024-04-19T02:03:03.738375+0000","last_undegraded":"2024-04-19T02:03:03.738375+0000","last_fullsized":"2024-04-19T02:03:03.738375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:02:58.486225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:02:58.486225+0000","last_clean_scrub_stamp":"2024-04-19T02:02:58.486225+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:20.275707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":19,"seq":81604378628,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":19,"seq":81604378627,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411331,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411332,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411332,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28196,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343644,"statfs":{"total":96636764160,"available":96607891456,"internally_reserved":0,"allocated":741376,"data_stored":606874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411332,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":18,"seq":77309411330,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411331,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":18,"seq":77309411331,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":85534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":17,"seq":73014444036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":17,"seq":73014444036,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":606874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":17,"seq":73014444036,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T02:03:06.168 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T02:03:06.446 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:03:06.447 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-19T02:03:06.463 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-04-19T02:03:06.205854+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":328276,"kb_used_data":3548,"kb_used_omap":18,"kb_used_meta":324077,"kb_avail":1132133804,"statfs":{"total":1159641169920,"available":1159305015296,"internally_reserved":0,"allocated":3633152,"data_stored":2031148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":331855161},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000623"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.738538+0000","last_change":"2024-04-19T02:03:02.570839+0000","last_active":"2024-04-19T02:03:03.738538+0000","last_peered":"2024-04-19T02:03:03.738538+0000","last_clean":"2024-04-19T02:03:03.738538+0000","last_became_active":"2024-04-19T02:03:02.570116+0000","last_became_peered":"2024-04-19T02:03:02.570116+0000","last_unstale":"2024-04-19T02:03:03.738538+0000","last_undegraded":"2024-04-19T02:03:03.738538+0000","last_fullsized":"2024-04-19T02:03:03.738538+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:23.134425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.556984+0000","last_change":"2024-04-19T02:03:02.988302+0000","last_active":"2024-04-19T02:03:03.556984+0000","last_peered":"2024-04-19T02:03:03.556984+0000","last_clean":"2024-04-19T02:03:03.556984+0000","last_became_active":"2024-04-19T02:03:02.988073+0000","last_became_peered":"2024-04-19T02:03:02.988073+0000","last_unstale":"2024-04-19T02:03:03.556984+0000","last_undegraded":"2024-04-19T02:03:03.556984+0000","last_fullsized":"2024-04-19T02:03:03.556984+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:28.455271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.884666+0000","last_change":"2024-04-19T02:03:02.595929+0000","last_active":"2024-04-19T02:03:03.884666+0000","last_peered":"2024-04-19T02:03:03.884666+0000","last_clean":"2024-04-19T02:03:03.884666+0000","last_became_active":"2024-04-19T02:03:02.595232+0000","last_became_peered":"2024-04-19T02:03:02.595232+0000","last_unstale":"2024-04-19T02:03:03.884666+0000","last_undegraded":"2024-04-19T02:03:03.884666+0000","last_fullsized":"2024-04-19T02:03:03.884666+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:04.077370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.556937+0000","last_change":"2024-04-19T02:03:02.597582+0000","last_active":"2024-04-19T02:03:03.556937+0000","last_peered":"2024-04-19T02:03:03.556937+0000","last_clean":"2024-04-19T02:03:03.556937+0000","last_became_active":"2024-04-19T02:03:02.597118+0000","last_became_peered":"2024-04-19T02:03:02.597118+0000","last_unstale":"2024-04-19T02:03:03.556937+0000","last_undegraded":"2024-04-19T02:03:03.556937+0000","last_fullsized":"2024-04-19T02:03:03.556937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:13.683143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.738491+0000","last_change":"2024-04-19T02:03:02.560133+0000","last_active":"2024-04-19T02:03:03.738491+0000","last_peered":"2024-04-19T02:03:03.738491+0000","last_clean":"2024-04-19T02:03:03.738491+0000","last_became_active":"2024-04-19T02:03:02.559377+0000","last_became_peered":"2024-04-19T02:03:02.559377+0000","last_unstale":"2024-04-19T02:03:03.738491+0000","last_undegraded":"2024-04-19T02:03:03.738491+0000","last_fullsized":"2024-04-19T02:03:03.738491+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:30.728400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:04.132226+0000","last_change":"2024-04-19T02:03:02.686352+0000","last_active":"2024-04-19T02:03:04.132226+0000","last_peered":"2024-04-19T02:03:04.132226+0000","last_clean":"2024-04-19T02:03:04.132226+0000","last_became_active":"2024-04-19T02:03:02.685552+0000","last_became_peered":"2024-04-19T02:03:02.685552+0000","last_unstale":"2024-04-19T02:03:04.132226+0000","last_undegraded":"2024-04-19T02:03:04.132226+0000","last_fullsized":"2024-04-19T02:03:04.132226+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:29.058838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.738484+0000","last_change":"2024-04-19T02:03:02.561076+0000","last_active":"2024-04-19T02:03:03.738484+0000","last_peered":"2024-04-19T02:03:03.738484+0000","last_clean":"2024-04-19T02:03:03.738484+0000","last_became_active":"2024-04-19T02:03:02.560210+0000","last_became_peered":"2024-04-19T02:03:02.560210+0000","last_unstale":"2024-04-19T02:03:03.738484+0000","last_undegraded":"2024-04-19T02:03:03.738484+0000","last_fullsized":"2024-04-19T02:03:03.738484+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:16.900004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.557982+0000","last_change":"2024-04-19T02:03:03.026806+0000","last_active":"2024-04-19T02:03:03.557982+0000","last_peered":"2024-04-19T02:03:03.557982+0000","last_clean":"2024-04-19T02:03:03.557982+0000","last_became_active":"2024-04-19T02:03:03.026063+0000","last_became_peered":"2024-04-19T02:03:03.026063+0000","last_unstale":"2024-04-19T02:03:03.557982+0000","last_undegraded":"2024-04-19T02:03:03.557982+0000","last_fullsized":"2024-04-19T02:03:03.557982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:19.673016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":191,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-19T02:03:03.738375+0000","last_change":"2024-04-19T02:02:59.974166+0000","last_active":"2024-04-19T02:03:03.738375+0000","last_peered":"2024-04-19T02:03:03.738375+0000","last_clean":"2024-04-19T02:03:03.738375+0000","last_became_active":"2024-04-19T02:02:59.972005+0000","last_became_peered":"2024-04-19T02:02:59.972005+0000","last_unstale":"2024-04-19T02:03:03.738375+0000","last_undegraded":"2024-04-19T02:03:03.738375+0000","last_fullsized":"2024-04-19T02:03:03.738375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:02:58.486225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:02:58.486225+0000","last_clean_scrub_stamp":"2024-04-19T02:02:58.486225+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:20.275707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":19,"seq":81604378629,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":85534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":19,"seq":81604378628,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":85534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411332,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":85534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411333,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":85534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411333,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28196,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343644,"statfs":{"total":96636764160,"available":96607891456,"internally_reserved":0,"allocated":741376,"data_stored":606874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411333,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":18,"seq":77309411331,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411332,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":18,"seq":77309411332,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":85534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":17,"seq":73014444036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":17,"seq":73014444036,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":606874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":17,"seq":73014444036,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":77946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T02:03:06.464 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-19T02:03:06.464 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-04-19T02:03:06.756 INFO:teuthology.orchestra.run.smithi059.stdout:epoch 23 2024-04-19T02:03:06.756 INFO:teuthology.orchestra.run.smithi059.stdout:fsid db9b8c36-b038-40a6-b749-a60ca7443e45 2024-04-19T02:03:06.756 INFO:teuthology.orchestra.run.smithi059.stdout:created 2024-04-19T02:02:47.352143+0000 2024-04-19T02:03:06.756 INFO:teuthology.orchestra.run.smithi059.stdout:modified 2024-04-19T02:03:03.551342+0000 2024-04-19T02:03:06.756 INFO:teuthology.orchestra.run.smithi059.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-19T02:03:06.756 INFO:teuthology.orchestra.run.smithi059.stdout:crush_version 5 2024-04-19T02:03:06.756 INFO:teuthology.orchestra.run.smithi059.stdout:full_ratio 0.95 2024-04-19T02:03:06.756 INFO:teuthology.orchestra.run.smithi059.stdout:backfillfull_ratio 0.9 2024-04-19T02:03:06.757 INFO:teuthology.orchestra.run.smithi059.stdout:nearfull_ratio 0.85 2024-04-19T02:03:06.757 INFO:teuthology.orchestra.run.smithi059.stdout:require_min_compat_client luminous 2024-04-19T02:03:06.757 INFO:teuthology.orchestra.run.smithi059.stdout:min_compat_client jewel 2024-04-19T02:03:06.757 INFO:teuthology.orchestra.run.smithi059.stdout:require_osd_release squid 2024-04-19T02:03:06.757 INFO:teuthology.orchestra.run.smithi059.stdout:stretch_mode_enabled false 2024-04-19T02:03:06.757 INFO:teuthology.orchestra.run.smithi059.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 20 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 11.76 2024-04-19T02:03:06.757 INFO:teuthology.orchestra.run.smithi059.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 23 flags hashpspool stripe_width 0 application rbd read_balance_score 4.52 2024-04-19T02:03:06.757 INFO:teuthology.orchestra.run.smithi059.stdout:max_osd 12 2024-04-19T02:03:06.757 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.59:6808/1145933874,v1:172.21.15.59:6809/1145933874] [v2:172.21.15.59:6810/1145933874,v1:172.21.15.59:6811/1145933874] exists,up dff2e9d5-c79f-4cc8-b68c-d1d35cc3dfeb 2024-04-19T02:03:06.757 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1 up in weight 1 up_from 17 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.59:6800/3807848039,v1:172.21.15.59:6801/3807848039] [v2:172.21.15.59:6802/3807848039,v1:172.21.15.59:6803/3807848039] exists,up 30cb7aa1-0bf8-4bc8-8bc5-f8b59870624e 2024-04-19T02:03:06.757 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2 up in weight 1 up_from 19 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.59:6816/420007638,v1:172.21.15.59:6817/420007638] [v2:172.21.15.59:6820/420007638,v1:172.21.15.59:6822/420007638] exists,up 2e8fca1b-e008-4e58-865e-09a658f6edd9 2024-04-19T02:03:06.757 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.59:6818/1791539182,v1:172.21.15.59:6819/1791539182] [v2:172.21.15.59:6821/1791539182,v1:172.21.15.59:6823/1791539182] exists,up 32289c9a-b7d9-4bcd-9f66-fa68f0c9796b 2024-04-19T02:03:06.757 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.81:6806/2921507000,v1:172.21.15.81:6808/2921507000] [v2:172.21.15.81:6810/2921507000,v1:172.21.15.81:6811/2921507000] exists,up 844daeea-860b-4853-a759-994531e5de3c 2024-04-19T02:03:06.757 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.81:6816/3925467179,v1:172.21.15.81:6818/3925467179] [v2:172.21.15.81:6819/3925467179,v1:172.21.15.81:6821/3925467179] exists,up f43ad412-74bb-43a6-95e7-25ee06bde025 2024-04-19T02:03:06.757 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.81:6800/3714790214,v1:172.21.15.81:6801/3714790214] [v2:172.21.15.81:6802/3714790214,v1:172.21.15.81:6803/3714790214] exists,up 07ccbc36-1f5d-4595-8578-09362b5e212f 2024-04-19T02:03:06.758 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7 up in weight 1 up_from 18 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.81:6817/2170932617,v1:172.21.15.81:6820/2170932617] [v2:172.21.15.81:6823/2170932617,v1:172.21.15.81:6827/2170932617] exists,up 6b2247e7-ebf8-4c0e-89ae-e15985f59aa1 2024-04-19T02:03:06.758 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8 up in weight 1 up_from 18 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6824/2053612068,v1:172.21.15.179:6825/2053612068] [v2:172.21.15.179:6826/2053612068,v1:172.21.15.179:6827/2053612068] exists,up 7142efd0-2144-4973-a01d-69a064b9b51f 2024-04-19T02:03:06.758 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6800/2641050728,v1:172.21.15.179:6801/2641050728] [v2:172.21.15.179:6802/2641050728,v1:172.21.15.179:6803/2641050728] exists,up 30a29baa-53cc-4c29-ab41-af14ead5c539 2024-04-19T02:03:06.758 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6816/3465727116,v1:172.21.15.179:6817/3465727116] [v2:172.21.15.179:6818/3465727116,v1:172.21.15.179:6819/3465727116] exists,up fae84343-8b7a-4bd3-aeef-823f2f6058e3 2024-04-19T02:03:06.758 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11 up in weight 1 up_from 18 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6808/4047209718,v1:172.21.15.179:6809/4047209718] [v2:172.21.15.179:6810/4047209718,v1:172.21.15.179:6811/4047209718] exists,up 35858134-6864-4da1-88c2-4b48166febab 2024-04-19T02:03:06.774 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.59:6808/1145933874,v1:172.21.15.59:6809/1145933874] [v2:172.21.15.59:6810/1145933874,v1:172.21.15.59:6811/1145933874] exists,up dff2e9d5-c79f-4cc8-b68c-d1d35cc3dfeb', 'osd.1 up in weight 1 up_from 17 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.59:6800/3807848039,v1:172.21.15.59:6801/3807848039] [v2:172.21.15.59:6802/3807848039,v1:172.21.15.59:6803/3807848039] exists,up 30cb7aa1-0bf8-4bc8-8bc5-f8b59870624e', 'osd.2 up in weight 1 up_from 19 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.59:6816/420007638,v1:172.21.15.59:6817/420007638] [v2:172.21.15.59:6820/420007638,v1:172.21.15.59:6822/420007638] exists,up 2e8fca1b-e008-4e58-865e-09a658f6edd9', 'osd.3 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.59:6818/1791539182,v1:172.21.15.59:6819/1791539182] [v2:172.21.15.59:6821/1791539182,v1:172.21.15.59:6823/1791539182] exists,up 32289c9a-b7d9-4bcd-9f66-fa68f0c9796b', 'osd.4 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.81:6806/2921507000,v1:172.21.15.81:6808/2921507000] [v2:172.21.15.81:6810/2921507000,v1:172.21.15.81:6811/2921507000] exists,up 844daeea-860b-4853-a759-994531e5de3c', 'osd.5 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.81:6816/3925467179,v1:172.21.15.81:6818/3925467179] [v2:172.21.15.81:6819/3925467179,v1:172.21.15.81:6821/3925467179] exists,up f43ad412-74bb-43a6-95e7-25ee06bde025', 'osd.6 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.81:6800/3714790214,v1:172.21.15.81:6801/3714790214] [v2:172.21.15.81:6802/3714790214,v1:172.21.15.81:6803/3714790214] exists,up 07ccbc36-1f5d-4595-8578-09362b5e212f', 'osd.7 up in weight 1 up_from 18 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.81:6817/2170932617,v1:172.21.15.81:6820/2170932617] [v2:172.21.15.81:6823/2170932617,v1:172.21.15.81:6827/2170932617] exists,up 6b2247e7-ebf8-4c0e-89ae-e15985f59aa1', 'osd.8 up in weight 1 up_from 18 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6824/2053612068,v1:172.21.15.179:6825/2053612068] [v2:172.21.15.179:6826/2053612068,v1:172.21.15.179:6827/2053612068] exists,up 7142efd0-2144-4973-a01d-69a064b9b51f', 'osd.9 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6800/2641050728,v1:172.21.15.179:6801/2641050728] [v2:172.21.15.179:6802/2641050728,v1:172.21.15.179:6803/2641050728] exists,up 30a29baa-53cc-4c29-ab41-af14ead5c539', 'osd.10 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6816/3465727116,v1:172.21.15.179:6817/3465727116] [v2:172.21.15.179:6818/3465727116,v1:172.21.15.179:6819/3465727116] exists,up fae84343-8b7a-4bd3-aeef-823f2f6058e3', 'osd.11 up in weight 1 up_from 18 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6808/4047209718,v1:172.21.15.179:6809/4047209718] [v2:172.21.15.179:6810/4047209718,v1:172.21.15.179:6811/4047209718] exists,up 35858134-6864-4da1-88c2-4b48166febab'] 2024-04-19T02:03:06.774 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-04-19T02:03:06.884 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:06.884 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "mon.a", 2024-04-19T02:03:06.884 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster": "ceph", 2024-04-19T02:03:06.884 INFO:teuthology.orchestra.run.smithi059.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-04-19T02:03:06.884 INFO:teuthology.orchestra.run.smithi059.stdout: "admin_socket_mode": "", 2024-04-19T02:03:06.884 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-04-19T02:03:06.885 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_client_required": "cephx, none", 2024-04-19T02:03:06.885 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_cluster_required": "cephx", 2024-04-19T02:03:06.885 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_debug": "true", 2024-04-19T02:03:06.885 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-04-19T02:03:06.885 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-04-19T02:03:06.885 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_service_required": "cephx", 2024-04-19T02:03:06.885 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_service_ticket_ttl": "240.000000", 2024-04-19T02:03:06.885 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_supported": "cephx", 2024-04-19T02:03:06.885 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_aio": "true", 2024-04-19T02:03:06.885 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_aio_max_queue_depth": "1024", 2024-04-19T02:03:06.885 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_aio_poll_ms": "250", 2024-04-19T02:03:06.885 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_aio_reap_max": "16", 2024-04-19T02:03:06.885 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_async_discard": "false", 2024-04-19T02:03:06.885 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_block_size": "4096", 2024-04-19T02:03:06.885 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_debug_aio": "false", 2024-04-19T02:03:06.886 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-04-19T02:03:06.886 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-04-19T02:03:06.886 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_debug_inflight_ios": "false", 2024-04-19T02:03:06.886 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_enable_discard": "false", 2024-04-19T02:03:06.886 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_flock_retry": "3", 2024-04-19T02:03:06.886 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_flock_retry_interval": "0.100000", 2024-04-19T02:03:06.886 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_inject_crash": "0", 2024-04-19T02:03:06.886 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_inject_crash_flush_delay": "2", 2024-04-19T02:03:06.886 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_ioring": "false", 2024-04-19T02:03:06.886 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_ioring_hipri": "false", 2024-04-19T02:03:06.886 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_ioring_sqthread_poll": "false", 2024-04-19T02:03:06.886 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-04-19T02:03:06.886 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_read_buffer_alignment": "4096", 2024-04-19T02:03:06.886 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-04-19T02:03:06.886 INFO:teuthology.orchestra.run.smithi059.stdout: "bdev_type": "", 2024-04-19T02:03:06.887 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_alloc_size": "1048576", 2024-04-19T02:03:06.887 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_allocator": "hybrid", 2024-04-19T02:03:06.887 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_buffered_io": "true", 2024-04-19T02:03:06.887 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_check_for_zeros": "false", 2024-04-19T02:03:06.887 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_check_volume_selector_often": "false", 2024-04-19T02:03:06.887 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-04-19T02:03:06.887 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_compact_log_sync": "false", 2024-04-19T02:03:06.887 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-04-19T02:03:06.887 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-04-19T02:03:06.887 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_log_compact_min_size": "16777216", 2024-04-19T02:03:06.887 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_log_replay_check_allocations": "true", 2024-04-19T02:03:06.887 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_max_log_runway": "4194304", 2024-04-19T02:03:06.887 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_max_prefetch": "1048576", 2024-04-19T02:03:06.887 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_min_flush_size": "524288", 2024-04-19T02:03:06.887 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_min_log_runway": "1048576", 2024-04-19T02:03:06.888 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_replay_recovery": "false", 2024-04-19T02:03:06.888 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-04-19T02:03:06.888 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_shared_alloc_size": "65536", 2024-04-19T02:03:06.888 INFO:teuthology.orchestra.run.smithi059.stdout: "bluefs_sync_write": "false", 2024-04-19T02:03:06.888 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-04-19T02:03:06.888 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-04-19T02:03:06.888 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-04-19T02:03:06.888 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_allocation_from_file": "true", 2024-04-19T02:03:06.888 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_allocator": "hybrid", 2024-04-19T02:03:06.888 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-04-19T02:03:06.888 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-04-19T02:03:06.888 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-04-19T02:03:06.888 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-04-19T02:03:06.888 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-04-19T02:03:06.889 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-04-19T02:03:06.889 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_blobid_prealloc": "10240", 2024-04-19T02:03:06.889 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_block_create": "true", 2024-04-19T02:03:06.889 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_block_db_create": "false", 2024-04-19T02:03:06.889 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_block_db_path": "", 2024-04-19T02:03:06.889 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_block_db_size": "0", 2024-04-19T02:03:06.889 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_block_path": "", 2024-04-19T02:03:06.889 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_block_preallocate_file": "false", 2024-04-19T02:03:06.889 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_block_size": "107374182400", 2024-04-19T02:03:06.889 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_block_wal_create": "false", 2024-04-19T02:03:06.889 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_block_wal_path": "", 2024-04-19T02:03:06.889 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_block_wal_size": "100663296", 2024-04-19T02:03:06.889 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_bluefs": "true", 2024-04-19T02:03:06.889 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-04-19T02:03:06.889 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_bluefs_env_mirror": "false", 2024-04-19T02:03:06.889 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-04-19T02:03:06.890 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-04-19T02:03:06.890 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-04-19T02:03:06.890 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-04-19T02:03:06.890 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-04-19T02:03:06.890 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-04-19T02:03:06.890 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_autotune": "true", 2024-04-19T02:03:06.890 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-04-19T02:03:06.890 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-04-19T02:03:06.890 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-04-19T02:03:06.890 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-04-19T02:03:06.890 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_size": "0", 2024-04-19T02:03:06.890 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-04-19T02:03:06.890 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-04-19T02:03:06.890 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-04-19T02:03:06.890 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-04-19T02:03:06.891 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cache_type": "2q", 2024-04-19T02:03:06.891 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-04-19T02:03:06.891 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_clone_cow": "true", 2024-04-19T02:03:06.891 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_compression_algorithm": "snappy", 2024-04-19T02:03:06.891 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_compression_max_blob_size": "0", 2024-04-19T02:03:06.891 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-04-19T02:03:06.891 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-04-19T02:03:06.891 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_compression_min_blob_size": "0", 2024-04-19T02:03:06.891 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-04-19T02:03:06.891 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-04-19T02:03:06.891 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_compression_mode": "none", 2024-04-19T02:03:06.891 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-04-19T02:03:06.891 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_csum_type": "crc32c", 2024-04-19T02:03:06.891 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_enforce_settings": "default", 2024-04-19T02:03:06.891 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_freelist": "false", 2024-04-19T02:03:06.891 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_fsck_abort": "false", 2024-04-19T02:03:06.891 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-04-19T02:03:06.892 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-04-19T02:03:06.892 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_inject_read_err": "false", 2024-04-19T02:03:06.892 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_legacy_omap": "false", 2024-04-19T02:03:06.892 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-04-19T02:03:06.892 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-04-19T02:03:06.892 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-04-19T02:03:06.892 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-04-19T02:03:06.892 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_prefill": "0.000000", 2024-04-19T02:03:06.892 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-04-19T02:03:06.892 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-04-19T02:03:06.892 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-04-19T02:03:06.892 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_small_allocations": "0", 2024-04-19T02:03:06.892 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-04-19T02:03:06.892 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_default_buffered_read": "true", 2024-04-19T02:03:06.892 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_default_buffered_write": "false", 2024-04-19T02:03:06.893 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_deferred_batch_ops": "0", 2024-04-19T02:03:06.893 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-04-19T02:03:06.893 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-04-19T02:03:06.893 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_elastic_shared_blobs": "true", 2024-04-19T02:03:06.893 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-04-19T02:03:06.893 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-04-19T02:03:06.893 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-04-19T02:03:06.893 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-04-19T02:03:06.893 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-04-19T02:03:06.893 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fail_eio": "false", 2024-04-19T02:03:06.893 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-04-19T02:03:06.893 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-04-19T02:03:06.893 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-04-19T02:03:06.893 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-04-19T02:03:06.893 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_on_mkfs": "true", 2024-04-19T02:03:06.894 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-04-19T02:03:06.894 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_on_mount": "false", 2024-04-19T02:03:06.894 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-04-19T02:03:06.894 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_on_umount": "false", 2024-04-19T02:03:06.894 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-04-19T02:03:06.894 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-04-19T02:03:06.894 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-04-19T02:03:06.894 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-04-19T02:03:06.894 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-04-19T02:03:06.894 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-04-19T02:03:06.894 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-04-19T02:03:06.894 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-04-19T02:03:06.894 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_ignore_data_csum": "false", 2024-04-19T02:03:06.894 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-04-19T02:03:06.894 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_kvbackend": "rocksdb", 2024-04-19T02:03:06.895 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-04-19T02:03:06.895 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-04-19T02:03:06.895 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_log_op_age": "5.000000", 2024-04-19T02:03:06.895 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_max_alloc_size": "0", 2024-04-19T02:03:06.895 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_max_blob_size": "0", 2024-04-19T02:03:06.895 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-04-19T02:03:06.895 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-04-19T02:03:06.895 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_max_defer_interval": "3.000000", 2024-04-19T02:03:06.895 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_max_deferred_txc": "32", 2024-04-19T02:03:06.895 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_min_alloc_size": "0", 2024-04-19T02:03:06.895 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-04-19T02:03:06.895 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-04-19T02:03:06.895 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_nid_prealloc": "1024", 2024-04-19T02:03:06.895 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_prefer_deferred_size": "0", 2024-04-19T02:03:06.895 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-04-19T02:03:06.895 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-04-19T02:03:06.895 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_qfsck_on_mount": "true", 2024-04-19T02:03:06.896 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_retry_disk_reads": "3", 2024-04-19T02:03:06.896 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_rocksdb_cf": "true", 2024-04-19T02:03:06.896 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-04-19T02:03:06.896 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-04-19T02:03:06.896 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_rocksdb_options_annex": "", 2024-04-19T02:03:06.896 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_spdk_coremask": "0x1", 2024-04-19T02:03:06.896 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_spdk_io_sleep": "5", 2024-04-19T02:03:06.896 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_spdk_max_io_completion": "0", 2024-04-19T02:03:06.896 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_spdk_mem": "512", 2024-04-19T02:03:06.896 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_sync_submit_transaction": "false", 2024-04-19T02:03:06.896 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_throttle_bytes": "67108864", 2024-04-19T02:03:06.896 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_throttle_cost_per_io": "0", 2024-04-19T02:03:06.896 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-04-19T02:03:06.896 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-04-19T02:03:06.896 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-04-19T02:03:06.897 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-04-19T02:03:06.897 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_tracing": "false", 2024-04-19T02:03:06.897 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-04-19T02:03:06.897 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-04-19T02:03:06.897 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_volume_selection_reserved": "0", 2024-04-19T02:03:06.897 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-04-19T02:03:06.897 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-04-19T02:03:06.897 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-04-19T02:03:06.897 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-04-19T02:03:06.897 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-04-19T02:03:06.897 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-04-19T02:03:06.897 INFO:teuthology.orchestra.run.smithi059.stdout: "bluestore_zero_block_detection": "false", 2024-04-19T02:03:06.897 INFO:teuthology.orchestra.run.smithi059.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-04-19T02:03:06.897 INFO:teuthology.orchestra.run.smithi059.stdout: "cephfs_mirror_action_update_interval": "2", 2024-04-19T02:03:06.897 INFO:teuthology.orchestra.run.smithi059.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-04-19T02:03:06.898 INFO:teuthology.orchestra.run.smithi059.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-04-19T02:03:06.898 INFO:teuthology.orchestra.run.smithi059.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-04-19T02:03:06.898 INFO:teuthology.orchestra.run.smithi059.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-04-19T02:03:06.898 INFO:teuthology.orchestra.run.smithi059.stdout: "cephfs_mirror_mount_timeout": "10", 2024-04-19T02:03:06.898 INFO:teuthology.orchestra.run.smithi059.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-04-19T02:03:06.898 INFO:teuthology.orchestra.run.smithi059.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-04-19T02:03:06.898 INFO:teuthology.orchestra.run.smithi059.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-04-19T02:03:06.898 INFO:teuthology.orchestra.run.smithi059.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-04-19T02:03:06.898 INFO:teuthology.orchestra.run.smithi059.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-04-19T02:03:06.898 INFO:teuthology.orchestra.run.smithi059.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-04-19T02:03:06.898 INFO:teuthology.orchestra.run.smithi059.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-04-19T02:03:06.898 INFO:teuthology.orchestra.run.smithi059.stdout: "cephx_cluster_require_signatures": "false", 2024-04-19T02:03:06.898 INFO:teuthology.orchestra.run.smithi059.stdout: "cephx_cluster_require_version": "2", 2024-04-19T02:03:06.898 INFO:teuthology.orchestra.run.smithi059.stdout: "cephx_require_signatures": "false", 2024-04-19T02:03:06.898 INFO:teuthology.orchestra.run.smithi059.stdout: "cephx_require_version": "2", 2024-04-19T02:03:06.899 INFO:teuthology.orchestra.run.smithi059.stdout: "cephx_service_require_signatures": "false", 2024-04-19T02:03:06.899 INFO:teuthology.orchestra.run.smithi059.stdout: "cephx_service_require_version": "2", 2024-04-19T02:03:06.899 INFO:teuthology.orchestra.run.smithi059.stdout: "cephx_sign_messages": "true", 2024-04-19T02:03:06.899 INFO:teuthology.orchestra.run.smithi059.stdout: "chdir": "", 2024-04-19T02:03:06.899 INFO:teuthology.orchestra.run.smithi059.stdout: "client_acl_type": "", 2024-04-19T02:03:06.899 INFO:teuthology.orchestra.run.smithi059.stdout: "client_asio_thread_count": "2", 2024-04-19T02:03:06.899 INFO:teuthology.orchestra.run.smithi059.stdout: "client_cache_mid": "0.750000", 2024-04-19T02:03:06.899 INFO:teuthology.orchestra.run.smithi059.stdout: "client_cache_size": "16384", 2024-04-19T02:03:06.899 INFO:teuthology.orchestra.run.smithi059.stdout: "client_caps_release_delay": "5", 2024-04-19T02:03:06.899 INFO:teuthology.orchestra.run.smithi059.stdout: "client_check_pool_perm": "true", 2024-04-19T02:03:06.899 INFO:teuthology.orchestra.run.smithi059.stdout: "client_collect_and_send_global_metrics": "false", 2024-04-19T02:03:06.899 INFO:teuthology.orchestra.run.smithi059.stdout: "client_debug_force_sync_read": "false", 2024-04-19T02:03:06.899 INFO:teuthology.orchestra.run.smithi059.stdout: "client_debug_getattr_caps": "false", 2024-04-19T02:03:06.899 INFO:teuthology.orchestra.run.smithi059.stdout: "client_debug_inject_tick_delay": "0", 2024-04-19T02:03:06.899 INFO:teuthology.orchestra.run.smithi059.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-04-19T02:03:06.899 INFO:teuthology.orchestra.run.smithi059.stdout: "client_die_on_failed_remount": "false", 2024-04-19T02:03:06.900 INFO:teuthology.orchestra.run.smithi059.stdout: "client_dirsize_rbytes": "true", 2024-04-19T02:03:06.900 INFO:teuthology.orchestra.run.smithi059.stdout: "client_force_lazyio": "false", 2024-04-19T02:03:06.900 INFO:teuthology.orchestra.run.smithi059.stdout: "client_fs": "", 2024-04-19T02:03:06.900 INFO:teuthology.orchestra.run.smithi059.stdout: "client_inject_fixed_oldest_tid": "false", 2024-04-19T02:03:06.900 INFO:teuthology.orchestra.run.smithi059.stdout: "client_inject_release_failure": "false", 2024-04-19T02:03:06.900 INFO:teuthology.orchestra.run.smithi059.stdout: "client_max_inline_size": "4096", 2024-04-19T02:03:06.900 INFO:teuthology.orchestra.run.smithi059.stdout: "client_max_retries_on_remount_failure": "5", 2024-04-19T02:03:06.900 INFO:teuthology.orchestra.run.smithi059.stdout: "client_mds_namespace": "", 2024-04-19T02:03:06.900 INFO:teuthology.orchestra.run.smithi059.stdout: "client_metadata": "", 2024-04-19T02:03:06.900 INFO:teuthology.orchestra.run.smithi059.stdout: "client_mount_gid": "-1", 2024-04-19T02:03:06.900 INFO:teuthology.orchestra.run.smithi059.stdout: "client_mount_timeout": "300", 2024-04-19T02:03:06.900 INFO:teuthology.orchestra.run.smithi059.stdout: "client_mount_uid": "-1", 2024-04-19T02:03:06.900 INFO:teuthology.orchestra.run.smithi059.stdout: "client_mountpoint": "/", 2024-04-19T02:03:06.900 INFO:teuthology.orchestra.run.smithi059.stdout: "client_notify_timeout": "10", 2024-04-19T02:03:06.900 INFO:teuthology.orchestra.run.smithi059.stdout: "client_oc": "true", 2024-04-19T02:03:06.900 INFO:teuthology.orchestra.run.smithi059.stdout: "client_oc_max_dirty": "104857600", 2024-04-19T02:03:06.900 INFO:teuthology.orchestra.run.smithi059.stdout: "client_oc_max_dirty_age": "5.000000", 2024-04-19T02:03:06.901 INFO:teuthology.orchestra.run.smithi059.stdout: "client_oc_max_objects": "1000", 2024-04-19T02:03:06.901 INFO:teuthology.orchestra.run.smithi059.stdout: "client_oc_size": "209715200", 2024-04-19T02:03:06.901 INFO:teuthology.orchestra.run.smithi059.stdout: "client_oc_target_dirty": "8388608", 2024-04-19T02:03:06.901 INFO:teuthology.orchestra.run.smithi059.stdout: "client_permissions": "true", 2024-04-19T02:03:06.901 INFO:teuthology.orchestra.run.smithi059.stdout: "client_quota": "true", 2024-04-19T02:03:06.901 INFO:teuthology.orchestra.run.smithi059.stdout: "client_quota_df": "true", 2024-04-19T02:03:06.901 INFO:teuthology.orchestra.run.smithi059.stdout: "client_readahead_max_bytes": "0", 2024-04-19T02:03:06.901 INFO:teuthology.orchestra.run.smithi059.stdout: "client_readahead_max_periods": "4", 2024-04-19T02:03:06.901 INFO:teuthology.orchestra.run.smithi059.stdout: "client_readahead_min": "131072", 2024-04-19T02:03:06.901 INFO:teuthology.orchestra.run.smithi059.stdout: "client_reconnect_stale": "false", 2024-04-19T02:03:06.901 INFO:teuthology.orchestra.run.smithi059.stdout: "client_shutdown_timeout": "30", 2024-04-19T02:03:06.901 INFO:teuthology.orchestra.run.smithi059.stdout: "client_snapdir": ".snap", 2024-04-19T02:03:06.901 INFO:teuthology.orchestra.run.smithi059.stdout: "client_tick_interval": "1", 2024-04-19T02:03:06.901 INFO:teuthology.orchestra.run.smithi059.stdout: "client_trace": "", 2024-04-19T02:03:06.901 INFO:teuthology.orchestra.run.smithi059.stdout: "client_try_dentry_invalidate": "false", 2024-04-19T02:03:06.901 INFO:teuthology.orchestra.run.smithi059.stdout: "client_use_faked_inos": "false", 2024-04-19T02:03:06.901 INFO:teuthology.orchestra.run.smithi059.stdout: "client_use_random_mds": "false", 2024-04-19T02:03:06.902 INFO:teuthology.orchestra.run.smithi059.stdout: "clog_to_graylog": "false", 2024-04-19T02:03:06.902 INFO:teuthology.orchestra.run.smithi059.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-04-19T02:03:06.902 INFO:teuthology.orchestra.run.smithi059.stdout: "clog_to_graylog_port": "12201", 2024-04-19T02:03:06.902 INFO:teuthology.orchestra.run.smithi059.stdout: "clog_to_monitors": "default=true", 2024-04-19T02:03:06.902 INFO:teuthology.orchestra.run.smithi059.stdout: "clog_to_syslog": "false", 2024-04-19T02:03:06.902 INFO:teuthology.orchestra.run.smithi059.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-04-19T02:03:06.902 INFO:teuthology.orchestra.run.smithi059.stdout: "clog_to_syslog_level": "info", 2024-04-19T02:03:06.902 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_addr": "-", 2024-04-19T02:03:06.902 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_network": "", 2024-04-19T02:03:06.902 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_network_interface": "", 2024-04-19T02:03:06.902 INFO:teuthology.orchestra.run.smithi059.stdout: "compressor_zlib_isal": "false", 2024-04-19T02:03:06.902 INFO:teuthology.orchestra.run.smithi059.stdout: "compressor_zlib_level": "5", 2024-04-19T02:03:06.902 INFO:teuthology.orchestra.run.smithi059.stdout: "compressor_zlib_winsize": "-15", 2024-04-19T02:03:06.902 INFO:teuthology.orchestra.run.smithi059.stdout: "compressor_zstd_level": "1", 2024-04-19T02:03:06.902 INFO:teuthology.orchestra.run.smithi059.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-04-19T02:03:06.903 INFO:teuthology.orchestra.run.smithi059.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-04-19T02:03:06.903 INFO:teuthology.orchestra.run.smithi059.stdout: "crimson_alien_op_num_threads": "6", 2024-04-19T02:03:06.903 INFO:teuthology.orchestra.run.smithi059.stdout: "crimson_alien_thread_cpu_cores": "", 2024-04-19T02:03:06.903 INFO:teuthology.orchestra.run.smithi059.stdout: "crimson_osd_obc_lru_size": "512", 2024-04-19T02:03:06.903 INFO:teuthology.orchestra.run.smithi059.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-04-19T02:03:06.903 INFO:teuthology.orchestra.run.smithi059.stdout: "crimson_seastar_cpu_cores": "", 2024-04-19T02:03:06.903 INFO:teuthology.orchestra.run.smithi059.stdout: "crush_location": "", 2024-04-19T02:03:06.903 INFO:teuthology.orchestra.run.smithi059.stdout: "crush_location_hook": "", 2024-04-19T02:03:06.903 INFO:teuthology.orchestra.run.smithi059.stdout: "crush_location_hook_timeout": "10", 2024-04-19T02:03:06.903 INFO:teuthology.orchestra.run.smithi059.stdout: "daemonize": "false", 2024-04-19T02:03:06.903 INFO:teuthology.orchestra.run.smithi059.stdout: "daos_pool": "tank", 2024-04-19T02:03:06.903 INFO:teuthology.orchestra.run.smithi059.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-04-19T02:03:06.903 INFO:teuthology.orchestra.run.smithi059.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-04-19T02:03:06.903 INFO:teuthology.orchestra.run.smithi059.stdout: "dbstore_db_name_prefix": "dbstore", 2024-04-19T02:03:06.903 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_alienstore": "0/5", 2024-04-19T02:03:06.904 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_allow_any_pool_priority": "false", 2024-04-19T02:03:06.904 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_asok": "1/5", 2024-04-19T02:03:06.904 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_asok_assert_abort": "false", 2024-04-19T02:03:06.904 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_asserts_on_shutdown": "true", 2024-04-19T02:03:06.904 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_auth": "20/20", 2024-04-19T02:03:06.904 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_bdev": "1/3", 2024-04-19T02:03:06.904 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_bluefs": "1/5", 2024-04-19T02:03:06.904 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_bluestore": "1/5", 2024-04-19T02:03:06.904 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_buffer": "0/1", 2024-04-19T02:03:06.904 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_ceph_exporter": "1/5", 2024-04-19T02:03:06.904 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_cephfs_mirror": "0/5", 2024-04-19T02:03:06.905 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_cephsqlite": "0/5", 2024-04-19T02:03:06.905 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_client": "0/5", 2024-04-19T02:03:06.905 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_compressor": "1/5", 2024-04-19T02:03:06.905 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_context": "0/1", 2024-04-19T02:03:06.905 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_crimson_interrupt": "0/5", 2024-04-19T02:03:06.905 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_crush": "1/1", 2024-04-19T02:03:06.905 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_crypto": "1/5", 2024-04-19T02:03:06.905 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_cyanstore": "0/5", 2024-04-19T02:03:06.905 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_deliberately_leak_memory": "false", 2024-04-19T02:03:06.905 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_disable_randomized_ping": "false", 2024-04-19T02:03:06.905 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_dpdk": "1/5", 2024-04-19T02:03:06.905 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_eventtrace": "1/5", 2024-04-19T02:03:06.905 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_filer": "0/1", 2024-04-19T02:03:06.905 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_filestore": "1/3", 2024-04-19T02:03:06.906 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_finisher": "1/1", 2024-04-19T02:03:06.906 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_fuse": "1/5", 2024-04-19T02:03:06.906 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_heartbeat_testing_span": "0", 2024-04-19T02:03:06.906 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_heartbeatmap": "1/5", 2024-04-19T02:03:06.906 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_immutable_obj_cache": "0/5", 2024-04-19T02:03:06.906 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_javaclient": "1/5", 2024-04-19T02:03:06.906 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_journal": "1/3", 2024-04-19T02:03:06.906 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_journaler": "0/5", 2024-04-19T02:03:06.906 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_kstore": "1/5", 2024-04-19T02:03:06.906 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_lockdep": "0/1", 2024-04-19T02:03:06.906 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mclock": "1/5", 2024-04-19T02:03:06.906 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mds": "1/5", 2024-04-19T02:03:06.906 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mds_balancer": "1/5", 2024-04-19T02:03:06.906 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mds_locker": "1/5", 2024-04-19T02:03:06.906 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mds_log": "1/5", 2024-04-19T02:03:06.907 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mds_log_expire": "1/5", 2024-04-19T02:03:06.907 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mds_migrator": "1/5", 2024-04-19T02:03:06.907 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mds_quiesce": "3/5", 2024-04-19T02:03:06.907 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_memstore": "1/5", 2024-04-19T02:03:06.907 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mgr": "2/5", 2024-04-19T02:03:06.907 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mgrc": "1/5", 2024-04-19T02:03:06.907 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_mon": "20/20", 2024-04-19T02:03:06.907 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_monc": "0/10", 2024-04-19T02:03:06.907 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_ms": "1/1", 2024-04-19T02:03:06.907 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_none": "0/5", 2024-04-19T02:03:06.907 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_objclass": "0/5", 2024-04-19T02:03:06.907 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_objectcacher": "0/5", 2024-04-19T02:03:06.907 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_objecter": "0/1", 2024-04-19T02:03:06.907 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_optracker": "0/5", 2024-04-19T02:03:06.907 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_osd": "1/5", 2024-04-19T02:03:06.908 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_paxos": "20/20", 2024-04-19T02:03:06.908 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_perfcounter": "1/5", 2024-04-19T02:03:06.908 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_prioritycache": "1/5", 2024-04-19T02:03:06.908 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rados": "0/5", 2024-04-19T02:03:06.908 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rbd": "0/5", 2024-04-19T02:03:06.908 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rbd_mirror": "0/5", 2024-04-19T02:03:06.908 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rbd_pwl": "0/5", 2024-04-19T02:03:06.908 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rbd_replay": "0/5", 2024-04-19T02:03:06.908 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_refs": "0/0", 2024-04-19T02:03:06.908 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_reserver": "1/1", 2024-04-19T02:03:06.908 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rgw": "1/5", 2024-04-19T02:03:06.908 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rgw_access": "1/5", 2024-04-19T02:03:06.908 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rgw_datacache": "1/5", 2024-04-19T02:03:06.908 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rgw_dbstore": "1/5", 2024-04-19T02:03:06.908 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rgw_flight": "1/5", 2024-04-19T02:03:06.908 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rgw_lifecycle": "1/5", 2024-04-19T02:03:06.909 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rgw_sync": "1/5", 2024-04-19T02:03:06.909 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_rocksdb": "4/5", 2024-04-19T02:03:06.909 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore": "0/5", 2024-04-19T02:03:06.909 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_backref": "0/5", 2024-04-19T02:03:06.909 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_cache": "0/5", 2024-04-19T02:03:06.909 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_cleaner": "0/5", 2024-04-19T02:03:06.909 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_device": "0/5", 2024-04-19T02:03:06.909 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_epm": "0/5", 2024-04-19T02:03:06.909 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-04-19T02:03:06.909 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_journal": "0/5", 2024-04-19T02:03:06.909 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_lba": "0/5", 2024-04-19T02:03:06.909 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_odata": "0/5", 2024-04-19T02:03:06.909 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_omap": "0/5", 2024-04-19T02:03:06.909 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_onode": "0/5", 2024-04-19T02:03:06.909 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_t": "0/5", 2024-04-19T02:03:06.910 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_seastore_tm": "0/5", 2024-04-19T02:03:06.910 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_striper": "0/1", 2024-04-19T02:03:06.910 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_test": "0/5", 2024-04-19T02:03:06.910 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_throttle": "1/1", 2024-04-19T02:03:06.910 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_timer": "0/1", 2024-04-19T02:03:06.910 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_tp": "0/5", 2024-04-19T02:03:06.910 INFO:teuthology.orchestra.run.smithi059.stdout: "debug_trace": "1/5", 2024-04-19T02:03:06.910 INFO:teuthology.orchestra.run.smithi059.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-04-19T02:03:06.910 INFO:teuthology.orchestra.run.smithi059.stdout: "device_failure_prediction_mode": "none", 2024-04-19T02:03:06.910 INFO:teuthology.orchestra.run.smithi059.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-04-19T02:03:06.910 INFO:teuthology.orchestra.run.smithi059.stdout: "erasure_code_dir": "/usr/lib/ceph/erasure-code", 2024-04-19T02:03:06.910 INFO:teuthology.orchestra.run.smithi059.stdout: "err_to_graylog": "false", 2024-04-19T02:03:06.910 INFO:teuthology.orchestra.run.smithi059.stdout: "err_to_journald": "false", 2024-04-19T02:03:06.910 INFO:teuthology.orchestra.run.smithi059.stdout: "err_to_stderr": "true", 2024-04-19T02:03:06.911 INFO:teuthology.orchestra.run.smithi059.stdout: "err_to_syslog": "false", 2024-04-19T02:03:06.911 INFO:teuthology.orchestra.run.smithi059.stdout: "event_tracing": "false", 2024-04-19T02:03:06.911 INFO:teuthology.orchestra.run.smithi059.stdout: "exporter_addr": "0.0.0.0", 2024-04-19T02:03:06.911 INFO:teuthology.orchestra.run.smithi059.stdout: "exporter_http_port": "9926", 2024-04-19T02:03:06.911 INFO:teuthology.orchestra.run.smithi059.stdout: "exporter_prio_limit": "5", 2024-04-19T02:03:06.911 INFO:teuthology.orchestra.run.smithi059.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-04-19T02:03:06.911 INFO:teuthology.orchestra.run.smithi059.stdout: "exporter_sort_metrics": "true", 2024-04-19T02:03:06.911 INFO:teuthology.orchestra.run.smithi059.stdout: "exporter_stats_period": "5", 2024-04-19T02:03:06.911 INFO:teuthology.orchestra.run.smithi059.stdout: "fake_statfs_for_testing": "0", 2024-04-19T02:03:06.911 INFO:teuthology.orchestra.run.smithi059.stdout: "fatal_signal_handlers": "true", 2024-04-19T02:03:06.911 INFO:teuthology.orchestra.run.smithi059.stdout: "filer_max_purge_ops": "10", 2024-04-19T02:03:06.911 INFO:teuthology.orchestra.run.smithi059.stdout: "filer_max_truncate_ops": "128", 2024-04-19T02:03:06.911 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_apply_finisher_threads": "1", 2024-04-19T02:03:06.911 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_blackhole": "false", 2024-04-19T02:03:06.911 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_btrfs_clone_range": "true", 2024-04-19T02:03:06.912 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_btrfs_snap": "true", 2024-04-19T02:03:06.912 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_caller_concurrency": "10", 2024-04-19T02:03:06.912 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_collect_device_partition_information": "true", 2024-04-19T02:03:06.912 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_commit_timeout": "600.000000", 2024-04-19T02:03:06.912 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_debug_inject_read_err": "false", 2024-04-19T02:03:06.912 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_debug_omap_check": "false", 2024-04-19T02:03:06.912 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_debug_random_read_err": "0.000000", 2024-04-19T02:03:06.912 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_debug_verify_split": "false", 2024-04-19T02:03:06.912 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_dump_file": "", 2024-04-19T02:03:06.912 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-04-19T02:03:06.912 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-04-19T02:03:06.912 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_fadvise": "true", 2024-04-19T02:03:06.912 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_fail_eio": "true", 2024-04-19T02:03:06.912 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_fd_cache_shards": "16", 2024-04-19T02:03:06.913 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_fd_cache_size": "128", 2024-04-19T02:03:06.913 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_fiemap": "false", 2024-04-19T02:03:06.913 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_fiemap_threshold": "4096", 2024-04-19T02:03:06.913 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-04-19T02:03:06.913 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_index_retry_probability": "0.000000", 2024-04-19T02:03:06.913 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_inject_stall": "0", 2024-04-19T02:03:06.913 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_journal_parallel": "false", 2024-04-19T02:03:06.913 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_journal_trailing": "false", 2024-04-19T02:03:06.913 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_journal_writeahead": "false", 2024-04-19T02:03:06.913 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_kill_at": "0", 2024-04-19T02:03:06.913 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-04-19T02:03:06.913 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_inline_xattr_size": "0", 2024-04-19T02:03:06.913 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-04-19T02:03:06.913 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-04-19T02:03:06.914 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-04-19T02:03:06.914 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_inline_xattrs": "0", 2024-04-19T02:03:06.914 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-04-19T02:03:06.914 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_inline_xattrs_other": "2", 2024-04-19T02:03:06.914 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-04-19T02:03:06.914 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_sync_interval": "5.000000", 2024-04-19T02:03:06.914 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_xattr_value_size": "0", 2024-04-19T02:03:06.914 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-04-19T02:03:06.914 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-04-19T02:03:06.914 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-04-19T02:03:06.914 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_merge_threshold": "-10", 2024-04-19T02:03:06.914 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_min_sync_interval": "0.010000", 2024-04-19T02:03:06.914 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_odsync_write": "false", 2024-04-19T02:03:06.914 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_omap_backend": "rocksdb", 2024-04-19T02:03:06.914 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_omap_backend_path": "", 2024-04-19T02:03:06.915 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_omap_header_cache_size": "1024", 2024-04-19T02:03:06.915 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_ondisk_finisher_threads": "1", 2024-04-19T02:03:06.915 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-04-19T02:03:06.915 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_op_thread_timeout": "60", 2024-04-19T02:03:06.915 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_op_threads": "2", 2024-04-19T02:03:06.915 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_punch_hole": "false", 2024-04-19T02:03:06.915 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-04-19T02:03:06.915 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-04-19T02:03:06.915 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-04-19T02:03:06.915 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-04-19T02:03:06.915 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-04-19T02:03:06.915 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_queue_max_bytes": "104857600", 2024-04-19T02:03:06.915 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-04-19T02:03:06.915 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-04-19T02:03:06.915 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-04-19T02:03:06.916 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_queue_max_ops": "50", 2024-04-19T02:03:06.916 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-04-19T02:03:06.916 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_seek_data_hole": "false", 2024-04-19T02:03:06.916 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_sloppy_crc": "false", 2024-04-19T02:03:06.916 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-04-19T02:03:06.916 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_splice": "false", 2024-04-19T02:03:06.916 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_split_multiple": "2", 2024-04-19T02:03:06.916 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_split_rand_factor": "20", 2024-04-19T02:03:06.916 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_update_to": "1000", 2024-04-19T02:03:06.916 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-04-19T02:03:06.916 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-04-19T02:03:06.916 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-04-19T02:03:06.916 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-04-19T02:03:06.916 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-04-19T02:03:06.916 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-04-19T02:03:06.917 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_enable": "true", 2024-04-19T02:03:06.917 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-04-19T02:03:06.917 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-04-19T02:03:06.917 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-04-19T02:03:06.917 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-04-19T02:03:06.917 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-04-19T02:03:06.917 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-04-19T02:03:06.917 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_xfs_extsize": "false", 2024-04-19T02:03:06.917 INFO:teuthology.orchestra.run.smithi059.stdout: "filestore_zfs_snap": "false", 2024-04-19T02:03:06.917 INFO:teuthology.orchestra.run.smithi059.stdout: "fio_dir": "/tmp/fio", 2024-04-19T02:03:06.917 INFO:teuthology.orchestra.run.smithi059.stdout: "fsid": "db9b8c36-b038-40a6-b749-a60ca7443e45", 2024-04-19T02:03:06.917 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_allow_other": "true", 2024-04-19T02:03:06.917 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_atomic_o_trunc": "true", 2024-04-19T02:03:06.917 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_big_writes": "true", 2024-04-19T02:03:06.917 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_debug": "false", 2024-04-19T02:03:06.918 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_default_permissions": "false", 2024-04-19T02:03:06.918 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_disable_pagecache": "false", 2024-04-19T02:03:06.918 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_max_write": "0", 2024-04-19T02:03:06.918 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_multithreaded": "true", 2024-04-19T02:03:06.918 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_require_active_mds": "true", 2024-04-19T02:03:06.918 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_set_user_groups": "true", 2024-04-19T02:03:06.918 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_splice_move": "true", 2024-04-19T02:03:06.918 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_splice_read": "true", 2024-04-19T02:03:06.918 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_splice_write": "true", 2024-04-19T02:03:06.918 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_syncfs_on_mksnap": "true", 2024-04-19T02:03:06.918 INFO:teuthology.orchestra.run.smithi059.stdout: "fuse_use_invalidate_cb": "true", 2024-04-19T02:03:06.918 INFO:teuthology.orchestra.run.smithi059.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-04-19T02:03:06.918 INFO:teuthology.orchestra.run.smithi059.stdout: "gss_target_name": "ceph", 2024-04-19T02:03:06.918 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_file": "", 2024-04-19T02:03:06.919 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_inject_failure": "0", 2024-04-19T02:03:06.919 INFO:teuthology.orchestra.run.smithi059.stdout: "heartbeat_interval": "5", 2024-04-19T02:03:06.919 INFO:teuthology.orchestra.run.smithi059.stdout: "host": "smithi059", 2024-04-19T02:03:06.919 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-04-19T02:03:06.919 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-04-19T02:03:06.919 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_max_size": "1073741824", 2024-04-19T02:03:06.919 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-04-19T02:03:06.919 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-04-19T02:03:06.919 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-04-19T02:03:06.919 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-04-19T02:03:06.919 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-04-19T02:03:06.919 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-04-19T02:03:06.919 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-04-19T02:03:06.919 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-04-19T02:03:06.919 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-04-19T02:03:06.920 INFO:teuthology.orchestra.run.smithi059.stdout: "immutable_object_cache_watermark": "0.900000", 2024-04-19T02:03:06.920 INFO:teuthology.orchestra.run.smithi059.stdout: "inject_early_sigterm": "false", 2024-04-19T02:03:06.920 INFO:teuthology.orchestra.run.smithi059.stdout: "jaeger_agent_port": "6799", 2024-04-19T02:03:06.920 INFO:teuthology.orchestra.run.smithi059.stdout: "jaeger_tracing_enable": "false", 2024-04-19T02:03:06.920 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_aio": "true", 2024-04-19T02:03:06.920 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_align_min_size": "65536", 2024-04-19T02:03:06.920 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_block_align": "true", 2024-04-19T02:03:06.920 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_block_size": "4096", 2024-04-19T02:03:06.920 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_dio": "true", 2024-04-19T02:03:06.920 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_discard": "false", 2024-04-19T02:03:06.920 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_force_aio": "false", 2024-04-19T02:03:06.920 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_ignore_corruption": "false", 2024-04-19T02:03:06.920 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_max_write_bytes": "10485760", 2024-04-19T02:03:06.920 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_max_write_entries": "100", 2024-04-19T02:03:06.920 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_replay_from": "0", 2024-04-19T02:03:06.920 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_throttle_high_multiple": "0.000000", 2024-04-19T02:03:06.921 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-04-19T02:03:06.921 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-04-19T02:03:06.921 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_throttle_max_multiple": "0.000000", 2024-04-19T02:03:06.921 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_write_header_frequency": "0", 2024-04-19T02:03:06.921 INFO:teuthology.orchestra.run.smithi059.stdout: "journal_zero_on_create": "false", 2024-04-19T02:03:06.921 INFO:teuthology.orchestra.run.smithi059.stdout: "journaler_prefetch_periods": "10", 2024-04-19T02:03:06.921 INFO:teuthology.orchestra.run.smithi059.stdout: "journaler_prezero_periods": "5", 2024-04-19T02:03:06.921 INFO:teuthology.orchestra.run.smithi059.stdout: "journaler_write_head_interval": "15", 2024-04-19T02:03:06.921 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "", 2024-04-19T02:03:06.921 INFO:teuthology.orchestra.run.smithi059.stdout: "keyfile": "", 2024-04-19T02:03:06.921 INFO:teuthology.orchestra.run.smithi059.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-04-19T02:03:06.921 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_backend": "rocksdb", 2024-04-19T02:03:06.921 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_default_stripe_size": "65536", 2024-04-19T02:03:06.921 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_fsck_on_mount": "false", 2024-04-19T02:03:06.921 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_fsck_on_mount_deep": "true", 2024-04-19T02:03:06.922 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_max_bytes": "67108864", 2024-04-19T02:03:06.922 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_max_ops": "512", 2024-04-19T02:03:06.922 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_nid_prealloc": "1024", 2024-04-19T02:03:06.922 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_onode_map_size": "1024", 2024-04-19T02:03:06.922 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-04-19T02:03:06.922 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_sync_submit_transaction": "false", 2024-04-19T02:03:06.922 INFO:teuthology.orchestra.run.smithi059.stdout: "kstore_sync_transaction": "false", 2024-04-19T02:03:06.922 INFO:teuthology.orchestra.run.smithi059.stdout: "librados_thread_count": "2", 2024-04-19T02:03:06.922 INFO:teuthology.orchestra.run.smithi059.stdout: "lockdep": "false", 2024-04-19T02:03:06.922 INFO:teuthology.orchestra.run.smithi059.stdout: "lockdep_force_backtrace": "false", 2024-04-19T02:03:06.922 INFO:teuthology.orchestra.run.smithi059.stdout: "log_coarse_timestamps": "true", 2024-04-19T02:03:06.922 INFO:teuthology.orchestra.run.smithi059.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-04-19T02:03:06.922 INFO:teuthology.orchestra.run.smithi059.stdout: "log_flush_on_exit": "false", 2024-04-19T02:03:06.922 INFO:teuthology.orchestra.run.smithi059.stdout: "log_graylog_host": "127.0.0.1", 2024-04-19T02:03:06.922 INFO:teuthology.orchestra.run.smithi059.stdout: "log_graylog_port": "12201", 2024-04-19T02:03:06.923 INFO:teuthology.orchestra.run.smithi059.stdout: "log_max_new": "1000", 2024-04-19T02:03:06.923 INFO:teuthology.orchestra.run.smithi059.stdout: "log_max_recent": "10000", 2024-04-19T02:03:06.923 INFO:teuthology.orchestra.run.smithi059.stdout: "log_stderr_prefix": "", 2024-04-19T02:03:06.923 INFO:teuthology.orchestra.run.smithi059.stdout: "log_stop_at_utilization": "0.970000", 2024-04-19T02:03:06.923 INFO:teuthology.orchestra.run.smithi059.stdout: "log_to_file": "true", 2024-04-19T02:03:06.923 INFO:teuthology.orchestra.run.smithi059.stdout: "log_to_graylog": "false", 2024-04-19T02:03:06.923 INFO:teuthology.orchestra.run.smithi059.stdout: "log_to_journald": "false", 2024-04-19T02:03:06.923 INFO:teuthology.orchestra.run.smithi059.stdout: "log_to_stderr": "false", 2024-04-19T02:03:06.923 INFO:teuthology.orchestra.run.smithi059.stdout: "log_to_syslog": "false", 2024-04-19T02:03:06.923 INFO:teuthology.orchestra.run.smithi059.stdout: "max_rotating_auth_attempts": "10", 2024-04-19T02:03:06.923 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-04-19T02:03:06.923 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_action_on_write_error": "1", 2024-04-19T02:03:06.923 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_alternate_name_max": "8192", 2024-04-19T02:03:06.923 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_asio_thread_count": "2", 2024-04-19T02:03:06.923 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_export_pin": "true", 2024-04-19T02:03:06.924 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_fragment_dirs": "true", 2024-04-19T02:03:06.924 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-04-19T02:03:06.924 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_fragment_interval": "5", 2024-04-19T02:03:06.924 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_fragment_size_max": "100000", 2024-04-19T02:03:06.924 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_idle_threshold": "0.000000", 2024-04-19T02:03:06.924 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_interval": "10", 2024-04-19T02:03:06.924 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_max": "-1", 2024-04-19T02:03:06.924 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_max_until": "-1", 2024-04-19T02:03:06.924 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_merge_size": "50", 2024-04-19T02:03:06.924 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_midchunk": "0.300000", 2024-04-19T02:03:06.924 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_min_rebalance": "0.100000", 2024-04-19T02:03:06.924 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_min_start": "0.200000", 2024-04-19T02:03:06.924 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_minchunk": "0.001000", 2024-04-19T02:03:06.924 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_mode": "0", 2024-04-19T02:03:06.924 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_need_max": "1.200000", 2024-04-19T02:03:06.924 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_need_min": "0.800000", 2024-04-19T02:03:06.924 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_overload_epochs": "2", 2024-04-19T02:03:06.925 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-04-19T02:03:06.925 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_sample_interval": "3.000000", 2024-04-19T02:03:06.925 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_split_bits": "3", 2024-04-19T02:03:06.925 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_split_rd": "25000.000000", 2024-04-19T02:03:06.925 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_split_size": "10000", 2024-04-19T02:03:06.925 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_split_wr": "10000.000000", 2024-04-19T02:03:06.925 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_target_decay": "10.000000", 2024-04-19T02:03:06.925 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-04-19T02:03:06.925 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_beacon_grace": "15.000000", 2024-04-19T02:03:06.925 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_beacon_interval": "4.000000", 2024-04-19T02:03:06.925 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_beacon_mon_down_grace": "60", 2024-04-19T02:03:06.925 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_memory_limit": "4294967296", 2024-04-19T02:03:06.925 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_mid": "0.700000", 2024-04-19T02:03:06.925 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-04-19T02:03:06.925 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_quiesce_delay": "0", 2024-04-19T02:03:06.926 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_quiesce_sleep": "200", 2024-04-19T02:03:06.926 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_quiesce_splitauth": "true", 2024-04-19T02:03:06.926 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_quiesce_threshold": "524288", 2024-04-19T02:03:06.926 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_release_free_interval": "10", 2024-04-19T02:03:06.926 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_reservation": "0.050000", 2024-04-19T02:03:06.926 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-04-19T02:03:06.926 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_trim_interval": "1", 2024-04-19T02:03:06.926 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cache_trim_threshold": "262144", 2024-04-19T02:03:06.926 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-04-19T02:03:06.926 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-04-19T02:03:06.926 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_client_delegate_inos_pct": "50", 2024-04-19T02:03:06.926 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_client_prealloc_inos": "1000", 2024-04-19T02:03:06.926 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-04-19T02:03:06.926 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_connect_bootstrapping": "false", 2024-04-19T02:03:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_damage_table_max_entries": "10000", 2024-04-19T02:03:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-04-19T02:03:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_debug_auth_pins": "false", 2024-04-19T02:03:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_debug_frag": "false", 2024-04-19T02:03:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_debug_scatterstat": "false", 2024-04-19T02:03:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_debug_subtrees": "false", 2024-04-19T02:03:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_decay_halflife": "5.000000", 2024-04-19T02:03:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_default_dir_hash": "2", 2024-04-19T02:03:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_defer_session_stale": "true", 2024-04-19T02:03:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_deny_all_reconnect": "false", 2024-04-19T02:03:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_dir_keys_per_op": "16384", 2024-04-19T02:03:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_dir_max_commit_size": "10", 2024-04-19T02:03:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_dir_max_entries": "0", 2024-04-19T02:03:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_dir_prefetch": "true", 2024-04-19T02:03:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_dirstat_min_interval": "1.000000", 2024-04-19T02:03:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_dump_cache_after_rejoin": "false", 2024-04-19T02:03:06.928 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_dump_cache_on_map": "false", 2024-04-19T02:03:06.928 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_dump_cache_threshold_file": "0", 2024-04-19T02:03:06.928 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-04-19T02:03:06.928 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_early_reply": "true", 2024-04-19T02:03:06.928 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_enable_op_tracker": "true", 2024-04-19T02:03:06.928 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_enforce_unique_name": "true", 2024-04-19T02:03:06.928 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_export_ephemeral_distributed": "true", 2024-04-19T02:03:06.928 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-04-19T02:03:06.928 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_export_ephemeral_random": "true", 2024-04-19T02:03:06.928 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-04-19T02:03:06.928 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-04-19T02:03:06.928 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_forward_all_requests_to_auth": "false", 2024-04-19T02:03:06.928 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-04-19T02:03:06.928 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-04-19T02:03:06.928 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-04-19T02:03:06.928 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-04-19T02:03:06.929 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_health_cache_threshold": "1.500000", 2024-04-19T02:03:06.929 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_health_summarize_threshold": "10", 2024-04-19T02:03:06.929 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_heartbeat_grace": "15.000000", 2024-04-19T02:03:06.929 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_heartbeat_reset_grace": "1000", 2024-04-19T02:03:06.929 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_inject_health_dummy": "false", 2024-04-19T02:03:06.929 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-04-19T02:03:06.929 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_inject_migrator_session_race": "false", 2024-04-19T02:03:06.929 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-04-19T02:03:06.929 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-04-19T02:03:06.929 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-04-19T02:03:06.929 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_join_fs": "", 2024-04-19T02:03:06.929 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_journal_format": "1", 2024-04-19T02:03:06.929 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_create_at": "0", 2024-04-19T02:03:06.929 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_export_at": "0", 2024-04-19T02:03:06.929 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_import_at": "0", 2024-04-19T02:03:06.929 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_journal_at": "0", 2024-04-19T02:03:06.930 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_journal_expire_at": "0", 2024-04-19T02:03:06.930 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_journal_replay_at": "0", 2024-04-19T02:03:06.930 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_link_at": "0", 2024-04-19T02:03:06.930 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_mdstable_at": "0", 2024-04-19T02:03:06.930 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_openc_at": "0", 2024-04-19T02:03:06.930 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_rename_at": "0", 2024-04-19T02:03:06.930 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_shutdown_at": "0", 2024-04-19T02:03:06.930 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-04-19T02:03:06.930 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_event_large_threshold": "524288", 2024-04-19T02:03:06.930 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_events_per_segment": "1024", 2024-04-19T02:03:06.930 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_major_segment_event_ratio": "12", 2024-04-19T02:03:06.930 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_max_events": "-1", 2024-04-19T02:03:06.930 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_max_segments": "128", 2024-04-19T02:03:06.930 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_pause": "false", 2024-04-19T02:03:06.931 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_segment_size": "0", 2024-04-19T02:03:06.931 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_skip_corrupt_events": "false", 2024-04-19T02:03:06.931 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_skip_unbounded_events": "false", 2024-04-19T02:03:06.931 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-04-19T02:03:06.931 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_trim_threshold": "128", 2024-04-19T02:03:06.931 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-04-19T02:03:06.931 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_log_warn_factor": "2.000000", 2024-04-19T02:03:06.931 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_max_caps_per_client": "1048576", 2024-04-19T02:03:06.931 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_max_completed_flushes": "100000", 2024-04-19T02:03:06.931 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_max_completed_requests": "100000", 2024-04-19T02:03:06.931 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_max_export_size": "20971520", 2024-04-19T02:03:06.931 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_max_file_recover": "32", 2024-04-19T02:03:06.931 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_max_purge_files": "64", 2024-04-19T02:03:06.931 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_max_purge_ops": "8192", 2024-04-19T02:03:06.931 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-04-19T02:03:06.931 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-04-19T02:03:06.932 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_max_snaps_per_dir": "100", 2024-04-19T02:03:06.932 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_metrics_update_interval": "2", 2024-04-19T02:03:06.932 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_min_caps_per_client": "100", 2024-04-19T02:03:06.932 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_min_caps_working_set": "10000", 2024-04-19T02:03:06.932 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-04-19T02:03:06.932 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_numa_node": "-1", 2024-04-19T02:03:06.932 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-04-19T02:03:06.932 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_op_complaint_time": "30.000000", 2024-04-19T02:03:06.932 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_op_history_duration": "600", 2024-04-19T02:03:06.932 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_op_history_size": "20", 2024-04-19T02:03:06.932 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_op_history_slow_op_size": "20", 2024-04-19T02:03:06.932 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_op_history_slow_op_threshold": "10", 2024-04-19T02:03:06.932 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_op_log_threshold": "5", 2024-04-19T02:03:06.932 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_ping_grace": "15", 2024-04-19T02:03:06.932 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_ping_interval": "5", 2024-04-19T02:03:06.932 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-04-19T02:03:06.933 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-04-19T02:03:06.933 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_recall_max_caps": "30000", 2024-04-19T02:03:06.933 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-04-19T02:03:06.933 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_recall_max_decay_threshold": "131072", 2024-04-19T02:03:06.933 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-04-19T02:03:06.933 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_recall_warning_threshold": "262144", 2024-04-19T02:03:06.933 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_reconnect_timeout": "45.000000", 2024-04-19T02:03:06.933 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_replay_interval": "1.000000", 2024-04-19T02:03:06.933 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-04-19T02:03:06.933 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-04-19T02:03:06.933 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_root_ino_gid": "0", 2024-04-19T02:03:06.933 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_root_ino_uid": "0", 2024-04-19T02:03:06.933 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-04-19T02:03:06.933 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_session_blocklist_on_evict": "true", 2024-04-19T02:03:06.933 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_session_blocklist_on_timeout": "true", 2024-04-19T02:03:06.933 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-04-19T02:03:06.934 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-04-19T02:03:06.934 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-04-19T02:03:06.934 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-04-19T02:03:06.934 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-04-19T02:03:06.934 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_session_metadata_threshold": "16777216", 2024-04-19T02:03:06.934 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-04-19T02:03:06.934 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_shutdown_check": "0", 2024-04-19T02:03:06.934 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_skip_ino": "0", 2024-04-19T02:03:06.934 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_sleep_rank_change": "0.000000", 2024-04-19T02:03:06.934 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_snap_max_uid": "4294967294", 2024-04-19T02:03:06.934 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_snap_min_uid": "0", 2024-04-19T02:03:06.934 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_snap_rstat": "false", 2024-04-19T02:03:06.934 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_standby_replay_damaged": "false", 2024-04-19T02:03:06.934 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_symlink_recovery": "true", 2024-04-19T02:03:06.934 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_task_status_update_interval": "2.000000", 2024-04-19T02:03:06.935 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_thrash_exports": "0", 2024-04-19T02:03:06.935 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_thrash_fragments": "0", 2024-04-19T02:03:06.935 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_tick_interval": "5.000000", 2024-04-19T02:03:06.935 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_valgrind_exit": "false", 2024-04-19T02:03:06.935 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_verify_backtrace": "1", 2024-04-19T02:03:06.935 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_verify_scatter": "false", 2024-04-19T02:03:06.935 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_wipe_ino_prealloc": "false", 2024-04-19T02:03:06.935 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_wipe_sessions": "false", 2024-04-19T02:03:06.935 INFO:teuthology.orchestra.run.smithi059.stdout: "mempool_debug": "false", 2024-04-19T02:03:06.935 INFO:teuthology.orchestra.run.smithi059.stdout: "memstore_debug_omit_block_device_write": "false", 2024-04-19T02:03:06.935 INFO:teuthology.orchestra.run.smithi059.stdout: "memstore_device_bytes": "1073741824", 2024-04-19T02:03:06.935 INFO:teuthology.orchestra.run.smithi059.stdout: "memstore_page_set": "false", 2024-04-19T02:03:06.935 INFO:teuthology.orchestra.run.smithi059.stdout: "memstore_page_size": "65536", 2024-04-19T02:03:06.935 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_client_bytes": "134217728", 2024-04-19T02:03:06.935 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_client_messages": "512", 2024-04-19T02:03:06.935 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-04-19T02:03:06.936 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_connect_retry_interval": "1.000000", 2024-04-19T02:03:06.936 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-04-19T02:03:06.936 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-04-19T02:03:06.936 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_disabled_modules": "", 2024-04-19T02:03:06.936 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-04-19T02:03:06.936 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_max_pg_num_change": "128", 2024-04-19T02:03:06.936 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_mds_bytes": "134217728", 2024-04-19T02:03:06.936 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_mds_messages": "128", 2024-04-19T02:03:06.936 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-04-19T02:03:06.936 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_mon_bytes": "134217728", 2024-04-19T02:03:06.936 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_mon_messages": "128", 2024-04-19T02:03:06.936 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_osd_bytes": "536870912", 2024-04-19T02:03:06.936 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_osd_messages": "8192", 2024-04-19T02:03:06.936 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_pool": "true", 2024-04-19T02:03:06.936 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_service_beacon_grace": "60.000000", 2024-04-19T02:03:06.936 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_standby_modules": "true", 2024-04-19T02:03:06.937 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_stats_period": "5", 2024-04-19T02:03:06.937 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_stats_threshold": "5", 2024-04-19T02:03:06.937 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_tick_period": "2", 2024-04-19T02:03:06.937 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-04-19T02:03:06.937 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_accept_timeout_factor": "2.000000", 2024-04-19T02:03:06.937 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_allow_pool_delete": "true", 2024-04-19T02:03:06.937 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_allow_pool_size_one": "true", 2024-04-19T02:03:06.937 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_auth_validate_all_caps": "true", 2024-04-19T02:03:06.937 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-04-19T02:03:06.937 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-04-19T02:03:06.937 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_bytes": "104857600", 2024-04-19T02:03:06.937 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_directed_command_retry": "5", 2024-04-19T02:03:06.937 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_hunt_interval": "3.000000", 2024-04-19T02:03:06.937 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-04-19T02:03:06.937 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-04-19T02:03:06.938 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-04-19T02:03:06.938 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_hunt_parallel": "3", 2024-04-19T02:03:06.938 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_log_interval": "1.000000", 2024-04-19T02:03:06.938 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-04-19T02:03:06.938 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_ping_interval": "10.000000", 2024-04-19T02:03:06.938 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_ping_timeout": "30.000000", 2024-04-19T02:03:06.938 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_client_target_rank": "-1", 2024-04-19T02:03:06.938 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_clock_drift_allowed": "1.000000", 2024-04-19T02:03:06.938 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-04-19T02:03:06.938 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-04-19T02:03:06.938 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_file_level": "debug", 2024-04-19T02:03:06.938 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_to_file": "true", 2024-04-19T02:03:06.938 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_to_graylog": "false", 2024-04-19T02:03:06.938 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-04-19T02:03:06.938 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-04-19T02:03:06.939 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_to_journald": "false", 2024-04-19T02:03:06.939 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_to_stderr": "false", 2024-04-19T02:03:06.939 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-04-19T02:03:06.939 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-04-19T02:03:06.939 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-04-19T02:03:06.939 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_compact_on_bootstrap": "false", 2024-04-19T02:03:06.939 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_compact_on_start": "false", 2024-04-19T02:03:06.939 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_compact_on_trim": "true", 2024-04-19T02:03:06.939 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_con_tracker_persist_interval": "10", 2024-04-19T02:03:06.939 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_con_tracker_score_halflife": "43200", 2024-04-19T02:03:06.939 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_config_key_max_entry_size": "65536", 2024-04-19T02:03:06.939 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_cpu_threads": "4", 2024-04-19T02:03:06.939 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_crush_min_required_version": "hammer", 2024-04-19T02:03:06.939 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_daemon_bytes": "419430400", 2024-04-19T02:03:06.939 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-04-19T02:03:06.939 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_data_avail_crit": "5", 2024-04-19T02:03:06.940 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_data_avail_warn": "5", 2024-04-19T02:03:06.940 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_data_size_warn": "16106127360", 2024-04-19T02:03:06.940 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_block_osdmap_trim": "false", 2024-04-19T02:03:06.940 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-04-19T02:03:06.940 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_dump_json": "false", 2024-04-19T02:03:06.940 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-04-19T02:03:06.940 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_dump_transactions": "false", 2024-04-19T02:03:06.940 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_extra_checks": "false", 2024-04-19T02:03:06.940 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-04-19T02:03:06.940 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-04-19T02:03:06.940 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_no_require_reef": "false", 2024-04-19T02:03:06.940 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_no_require_squid": "false", 2024-04-19T02:03:06.940 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-04-19T02:03:06.940 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_delta_reset_interval": "10.000000", 2024-04-19T02:03:06.940 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_dns_srv_name": "ceph-mon", 2024-04-19T02:03:06.941 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_down_mkfs_grace": "120", 2024-04-19T02:03:06.941 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_down_uptime_grace": "60", 2024-04-19T02:03:06.941 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_election_default_strategy": "1", 2024-04-19T02:03:06.941 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_election_timeout": "5.000000", 2024-04-19T02:03:06.941 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-04-19T02:03:06.941 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_elector_ping_divisor": "2", 2024-04-19T02:03:06.941 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_elector_ping_timeout": "2.000000", 2024-04-19T02:03:06.941 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_enable_op_tracker": "true", 2024-04-19T02:03:06.941 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_fake_pool_delete": "false", 2024-04-19T02:03:06.941 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_force_quorum_join": "false", 2024-04-19T02:03:06.941 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_globalid_prealloc": "10000", 2024-04-19T02:03:06.941 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_health_detail_to_clog": "false", 2024-04-19T02:03:06.941 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_health_log_update_period": "5", 2024-04-19T02:03:06.941 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_health_max_detail": "50", 2024-04-19T02:03:06.941 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_health_to_clog": "true", 2024-04-19T02:03:06.941 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_health_to_clog_interval": "600", 2024-04-19T02:03:06.942 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-04-19T02:03:06.942 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_host": "172.21.15.59,172.21.15.81,172.21.15.179", 2024-04-19T02:03:06.942 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_host_override": "", 2024-04-19T02:03:06.942 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_initial_members": "", 2024-04-19T02:03:06.942 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-04-19T02:03:06.942 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-04-19T02:03:06.942 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-04-19T02:03:06.942 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-04-19T02:03:06.942 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_keyvaluedb": "rocksdb", 2024-04-19T02:03:06.942 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_lease": "5.000000", 2024-04-19T02:03:06.942 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-04-19T02:03:06.942 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-04-19T02:03:06.942 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_log_full_interval": "50", 2024-04-19T02:03:06.942 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_log_max": "10000", 2024-04-19T02:03:06.943 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_log_max_summary": "50", 2024-04-19T02:03:06.943 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_max_log_entries_per_event": "4096", 2024-04-19T02:03:06.943 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_max_log_epochs": "500", 2024-04-19T02:03:06.943 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_max_mdsmap_epochs": "500", 2024-04-19T02:03:06.943 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_max_mgrmap_epochs": "500", 2024-04-19T02:03:06.943 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_max_osd": "10000", 2024-04-19T02:03:06.943 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_max_pg_per_osd": "10000", 2024-04-19T02:03:06.943 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_max_pool_pg_num": "65536", 2024-04-19T02:03:06.943 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-04-19T02:03:06.943 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-04-19T02:03:06.943 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_mds_force_trim_to": "0", 2024-04-19T02:03:06.943 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_mds_skip_sanity": "false", 2024-04-19T02:03:06.943 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_memory_autotune": "true", 2024-04-19T02:03:06.943 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_memory_target": "2147483648", 2024-04-19T02:03:06.943 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_mgr_beacon_grace": "30", 2024-04-19T02:03:06.943 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-04-19T02:03:06.944 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_mgr_digest_period": "5", 2024-04-19T02:03:06.944 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_mgr_inactive_grace": "60", 2024-04-19T02:03:06.944 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_mgr_mkfs_grace": "240", 2024-04-19T02:03:06.944 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-04-19T02:03:06.944 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_min_osdmap_epochs": "500", 2024-04-19T02:03:06.944 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_op_complaint_time": "30", 2024-04-19T02:03:06.944 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_op_history_duration": "600", 2024-04-19T02:03:06.944 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_op_history_size": "20", 2024-04-19T02:03:06.944 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_op_history_slow_op_size": "20", 2024-04-19T02:03:06.944 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_op_history_slow_op_threshold": "10", 2024-04-19T02:03:06.944 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_op_log_threshold": "5", 2024-04-19T02:03:06.944 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-04-19T02:03:06.944 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-04-19T02:03:06.944 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-04-19T02:03:06.944 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_auto_mark_in": "false", 2024-04-19T02:03:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_auto_mark_new_in": "true", 2024-04-19T02:03:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-04-19T02:03:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-04-19T02:03:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_cache_size": "500", 2024-04-19T02:03:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_cache_size_min": "134217728", 2024-04-19T02:03:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_crush_smoke_test": "true", 2024-04-19T02:03:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_destroyed_out_interval": "600", 2024-04-19T02:03:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_down_out_interval": "600", 2024-04-19T02:03:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-04-19T02:03:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-04-19T02:03:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_force_trim_to": "0", 2024-04-19T02:03:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_full_ratio": "0.950000", 2024-04-19T02:03:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-04-19T02:03:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_laggy_halflife": "3600", 2024-04-19T02:03:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_laggy_max_interval": "300", 2024-04-19T02:03:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_laggy_weight": "0.300000", 2024-04-19T02:03:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-04-19T02:03:06.945 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_max_creating_pgs": "1024", 2024-04-19T02:03:06.946 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_max_initial_pgs": "1024", 2024-04-19T02:03:06.946 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_min_down_reporters": "2", 2024-04-19T02:03:06.946 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-04-19T02:03:06.946 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-04-19T02:03:06.946 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-04-19T02:03:06.946 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_prime_pg_temp": "true", 2024-04-19T02:03:06.946 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-04-19T02:03:06.946 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-04-19T02:03:06.946 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_report_timeout": "900", 2024-04-19T02:03:06.946 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-04-19T02:03:06.946 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-04-19T02:03:06.946 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_warn_num_repaired": "10", 2024-04-19T02:03:06.946 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osd_warn_op_age": "32.000000", 2024-04-19T02:03:06.946 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-04-19T02:03:06.946 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osdmap_full_prune_interval": "10", 2024-04-19T02:03:06.947 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osdmap_full_prune_min": "10000", 2024-04-19T02:03:06.947 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-04-19T02:03:06.947 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-04-19T02:03:06.947 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_pg_stuck_threshold": "60", 2024-04-19T02:03:06.947 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-04-19T02:03:06.947 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_pg_warn_min_objects": "10000", 2024-04-19T02:03:06.947 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_pg_warn_min_per_osd": "0", 2024-04-19T02:03:06.947 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-04-19T02:03:06.947 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_pool_quota_crit_threshold": "0", 2024-04-19T02:03:06.947 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_pool_quota_warn_threshold": "0", 2024-04-19T02:03:06.947 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_probe_timeout": "2.000000", 2024-04-19T02:03:06.947 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_reweight_max_change": "0.050000", 2024-04-19T02:03:06.947 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_reweight_max_osds": "4", 2024-04-19T02:03:06.947 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-04-19T02:03:06.947 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-04-19T02:03:06.948 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-04-19T02:03:06.948 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-04-19T02:03:06.948 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-04-19T02:03:06.948 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_scrub_interval": "300", 2024-04-19T02:03:06.948 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_scrub_max_keys": "100", 2024-04-19T02:03:06.948 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_scrub_timeout": "300", 2024-04-19T02:03:06.948 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_session_timeout": "300", 2024-04-19T02:03:06.948 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_smart_report_timeout": "5", 2024-04-19T02:03:06.948 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_stat_smooth_intervals": "6", 2024-04-19T02:03:06.948 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-04-19T02:03:06.948 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_stretch_pool_min_size": "2", 2024-04-19T02:03:06.948 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_stretch_pool_size": "4", 2024-04-19T02:03:06.948 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-04-19T02:03:06.948 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_subscribe_interval": "86400.000000", 2024-04-19T02:03:06.948 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_sync_debug": "false", 2024-04-19T02:03:06.948 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_sync_max_payload_keys": "2000", 2024-04-19T02:03:06.949 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_sync_max_payload_size": "1048576", 2024-04-19T02:03:06.949 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_sync_provider_kill_at": "0", 2024-04-19T02:03:06.949 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_sync_requester_kill_at": "0", 2024-04-19T02:03:06.949 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_sync_timeout": "60.000000", 2024-04-19T02:03:06.949 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_target_pg_per_osd": "100", 2024-04-19T02:03:06.949 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_tick_interval": "5", 2024-04-19T02:03:06.949 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_timecheck_interval": "300.000000", 2024-04-19T02:03:06.949 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-04-19T02:03:06.949 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_use_min_delay_socket": "false", 2024-04-19T02:03:06.949 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_older_version_delay": "604800", 2024-04-19T02:03:06.949 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-04-19T02:03:06.949 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-04-19T02:03:06.949 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-04-19T02:03:06.949 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_filestore_osds": "false", 2024-04-19T02:03:06.949 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-04-19T02:03:06.949 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-04-19T02:03:06.949 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-04-19T02:03:06.950 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_misplaced": "false", 2024-04-19T02:03:06.950 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-04-19T02:03:06.950 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_older_version": "true", 2024-04-19T02:03:06.950 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-04-19T02:03:06.950 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_pool_no_app": "true", 2024-04-19T02:03:06.950 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-04-19T02:03:06.950 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-04-19T02:03:06.950 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-04-19T02:03:06.950 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-04-19T02:03:06.950 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_on_too_few_osds": "false", 2024-04-19T02:03:06.950 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-04-19T02:03:06.950 INFO:teuthology.orchestra.run.smithi059.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-04-19T02:03:06.950 INFO:teuthology.orchestra.run.smithi059.stdout: "monmap": "", 2024-04-19T02:03:06.950 INFO:teuthology.orchestra.run.smithi059.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-19T02:03:06.950 INFO:teuthology.orchestra.run.smithi059.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-04-19T02:03:06.950 INFO:teuthology.orchestra.run.smithi059.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-04-19T02:03:06.951 INFO:teuthology.orchestra.run.smithi059.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-19T02:03:06.951 INFO:teuthology.orchestra.run.smithi059.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-04-19T02:03:06.951 INFO:teuthology.orchestra.run.smithi059.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-04-19T02:03:06.951 INFO:teuthology.orchestra.run.smithi059.stdout: "motr_tracing_enabled": "false", 2024-04-19T02:03:06.951 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_op_threads": "3", 2024-04-19T02:03:06.951 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_buffer_size": "131072", 2024-04-19T02:03:06.951 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_cm": "false", 2024-04-19T02:03:06.951 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_device_name": "", 2024-04-19T02:03:06.951 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_dscp": "96", 2024-04-19T02:03:06.951 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-04-19T02:03:06.951 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_gid_idx": "0", 2024-04-19T02:03:06.951 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_local_gid": "", 2024-04-19T02:03:06.951 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_polling_us": "1000", 2024-04-19T02:03:06.951 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_port_num": "1", 2024-04-19T02:03:06.951 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-04-19T02:03:06.951 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-04-19T02:03:06.952 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_roce_ver": "1", 2024-04-19T02:03:06.952 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_send_buffers": "1024", 2024-04-19T02:03:06.952 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_sl": "3", 2024-04-19T02:03:06.952 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_support_srq": "true", 2024-04-19T02:03:06.952 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_rdma_type": "ib", 2024-04-19T02:03:06.952 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_async_reap_threshold": "5", 2024-04-19T02:03:06.952 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_bind_before_connect": "false", 2024-04-19T02:03:06.952 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_bind_ipv4": "true", 2024-04-19T02:03:06.952 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_bind_ipv6": "false", 2024-04-19T02:03:06.952 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_bind_msgr1": "true", 2024-04-19T02:03:06.952 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_bind_msgr2": "true", 2024-04-19T02:03:06.952 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_bind_port_max": "7568", 2024-04-19T02:03:06.952 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_bind_port_min": "6800", 2024-04-19T02:03:06.952 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_bind_prefer_ipv4": "false", 2024-04-19T02:03:06.952 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_bind_retry_count": "3", 2024-04-19T02:03:06.952 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_bind_retry_delay": "5", 2024-04-19T02:03:06.953 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_blackhole_client": "false", 2024-04-19T02:03:06.953 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_blackhole_mds": "false", 2024-04-19T02:03:06.953 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_blackhole_mgr": "false", 2024-04-19T02:03:06.953 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_blackhole_mon": "false", 2024-04-19T02:03:06.953 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_blackhole_osd": "false", 2024-04-19T02:03:06.953 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_client_mode": "crc secure", 2024-04-19T02:03:06.953 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_cluster_mode": "crc secure", 2024-04-19T02:03:06.953 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_cluster_type": "", 2024-04-19T02:03:06.953 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_compress_secure": "false", 2024-04-19T02:03:06.953 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_connection_idle_timeout": "900", 2024-04-19T02:03:06.953 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_connection_ready_timeout": "10", 2024-04-19T02:03:06.953 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_crc_data": "true", 2024-04-19T02:03:06.953 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_crc_header": "true", 2024-04-19T02:03:06.953 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_die_on_bad_msg": "false", 2024-04-19T02:03:06.953 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_die_on_bug": "true", 2024-04-19T02:03:06.953 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_die_on_old_message": "true", 2024-04-19T02:03:06.954 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_die_on_skipped_message": "false", 2024-04-19T02:03:06.954 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_die_on_unhandled_msg": "false", 2024-04-19T02:03:06.954 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-04-19T02:03:06.954 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_coremask": "0xF", 2024-04-19T02:03:06.954 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-04-19T02:03:06.954 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_devs_allowlist": "", 2024-04-19T02:03:06.954 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_enable_tso": "true", 2024-04-19T02:03:06.954 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-04-19T02:03:06.954 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-04-19T02:03:06.954 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_hugepages": "", 2024-04-19T02:03:06.954 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_hw_flow_control": "true", 2024-04-19T02:03:06.954 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-04-19T02:03:06.954 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_lro": "true", 2024-04-19T02:03:06.954 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_memory_channel": "4", 2024-04-19T02:03:06.955 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-04-19T02:03:06.955 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_pmd": "", 2024-04-19T02:03:06.955 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_port_id": "0", 2024-04-19T02:03:06.955 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-04-19T02:03:06.955 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dump_corrupt_message_level": "1", 2024-04-19T02:03:06.955 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_dump_on_send": "false", 2024-04-19T02:03:06.955 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_initial_backoff": "0.200000", 2024-04-19T02:03:06.955 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_inject_delay_max": "1.000000", 2024-04-19T02:03:06.955 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_inject_delay_probability": "0.000000", 2024-04-19T02:03:06.955 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_inject_delay_type": "", 2024-04-19T02:03:06.955 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_inject_internal_delays": "0.000000", 2024-04-19T02:03:06.955 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_inject_network_congestion": "0", 2024-04-19T02:03:06.955 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_inject_socket_failures": "5000", 2024-04-19T02:03:06.955 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_learn_addr_from_peer": "true", 2024-04-19T02:03:06.955 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_max_accept_failures": "4", 2024-04-19T02:03:06.955 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_max_backoff": "15.000000", 2024-04-19T02:03:06.956 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_mon_client_mode": "secure crc", 2024-04-19T02:03:06.956 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_mon_cluster_mode": "secure crc", 2024-04-19T02:03:06.956 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_mon_service_mode": "secure crc", 2024-04-19T02:03:06.956 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_osd_compress_min_size": "1024", 2024-04-19T02:03:06.956 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_osd_compress_mode": "none", 2024-04-19T02:03:06.956 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_osd_compression_algorithm": "snappy", 2024-04-19T02:03:06.956 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-04-19T02:03:06.956 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_pq_min_cost": "65536", 2024-04-19T02:03:06.956 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_public_type": "", 2024-04-19T02:03:06.956 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_service_mode": "crc secure", 2024-04-19T02:03:06.956 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_tcp_listen_backlog": "512", 2024-04-19T02:03:06.956 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_tcp_nodelay": "true", 2024-04-19T02:03:06.956 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-04-19T02:03:06.956 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_tcp_rcvbuf": "0", 2024-04-19T02:03:06.957 INFO:teuthology.orchestra.run.smithi059.stdout: "ms_type": "async+posix", 2024-04-19T02:03:06.957 INFO:teuthology.orchestra.run.smithi059.stdout: "no_config_file": "false", 2024-04-19T02:03:06.957 INFO:teuthology.orchestra.run.smithi059.stdout: "objecter_completion_locks_per_session": "32", 2024-04-19T02:03:06.957 INFO:teuthology.orchestra.run.smithi059.stdout: "objecter_debug_inject_relock_delay": "false", 2024-04-19T02:03:06.957 INFO:teuthology.orchestra.run.smithi059.stdout: "objecter_inflight_op_bytes": "104857600", 2024-04-19T02:03:06.957 INFO:teuthology.orchestra.run.smithi059.stdout: "objecter_inflight_ops": "1024", 2024-04-19T02:03:06.957 INFO:teuthology.orchestra.run.smithi059.stdout: "objecter_inject_no_watch_ping": "false", 2024-04-19T02:03:06.957 INFO:teuthology.orchestra.run.smithi059.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-04-19T02:03:06.957 INFO:teuthology.orchestra.run.smithi059.stdout: "objecter_tick_interval": "5.000000", 2024-04-19T02:03:06.957 INFO:teuthology.orchestra.run.smithi059.stdout: "objecter_timeout": "10.000000", 2024-04-19T02:03:06.957 INFO:teuthology.orchestra.run.smithi059.stdout: "objectstore_blackhole": "false", 2024-04-19T02:03:06.957 INFO:teuthology.orchestra.run.smithi059.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-04-19T02:03:06.957 INFO:teuthology.orchestra.run.smithi059.stdout: "openssl_engine_opts": "", 2024-04-19T02:03:06.957 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_agent_delay_time": "5.000000", 2024-04-19T02:03:06.957 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_agent_hist_halflife": "1000", 2024-04-19T02:03:06.957 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_agent_max_low_ops": "2", 2024-04-19T02:03:06.958 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_agent_max_ops": "4", 2024-04-19T02:03:06.958 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-04-19T02:03:06.958 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_agent_quantize_effort": "0.100000", 2024-04-19T02:03:06.958 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_agent_slop": "0.020000", 2024-04-19T02:03:06.958 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-04-19T02:03:06.958 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_allow_recovery_below_min_size": "true", 2024-04-19T02:03:06.958 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_asio_thread_count": "2", 2024-04-19T02:03:06.958 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_async_recovery_min_cost": "100", 2024-04-19T02:03:06.958 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_auto_mark_unfound_lost": "false", 2024-04-19T02:03:06.958 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_backfill_retry_interval": "30.000000", 2024-04-19T02:03:06.958 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_backfill_scan_max": "512", 2024-04-19T02:03:06.958 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_backfill_scan_min": "64", 2024-04-19T02:03:06.958 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_backoff_on_degraded": "false", 2024-04-19T02:03:06.958 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_backoff_on_peering": "false", 2024-04-19T02:03:06.958 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_backoff_on_unfound": "true", 2024-04-19T02:03:06.959 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_beacon_report_interval": "300", 2024-04-19T02:03:06.959 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_bench_duration": "30", 2024-04-19T02:03:06.959 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-04-19T02:03:06.959 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_bench_max_block_size": "67108864", 2024-04-19T02:03:06.959 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_bench_small_size_max_iops": "100", 2024-04-19T02:03:06.959 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_blkin_trace_all": "false", 2024-04-19T02:03:06.959 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_blocked_scrub_grace_period": "120", 2024-04-19T02:03:06.959 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-04-19T02:03:06.959 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-04-19T02:03:06.959 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-04-19T02:03:06.959 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_check_for_log_corruption": "false", 2024-04-19T02:03:06.959 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-04-19T02:03:06.959 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-19T02:03:06.959 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_class_dir": "/usr/lib/rados-classes", 2024-04-19T02:03:06.960 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-19T02:03:06.960 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_class_update_on_start": "true", 2024-04-19T02:03:06.960 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_client_message_cap": "256", 2024-04-19T02:03:06.960 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_client_message_size_cap": "524288000", 2024-04-19T02:03:06.960 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_client_op_priority": "63", 2024-04-19T02:03:06.960 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_client_watch_timeout": "30", 2024-04-19T02:03:06.960 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_command_max_records": "256", 2024-04-19T02:03:06.960 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_compact_on_start": "false", 2024-04-19T02:03:06.960 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-04-19T02:03:06.960 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_crush_chooseleaf_type": "0", 2024-04-19T02:03:06.960 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_crush_initial_weight": "-1.000000", 2024-04-19T02:03:06.960 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_crush_update_on_start": "true", 2024-04-19T02:03:06.960 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_crush_update_weight_set": "true", 2024-04-19T02:03:06.960 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-04-19T02:03:06.961 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-04-19T02:03:06.961 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-04-19T02:03:06.961 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_drop_ping_duration": "0", 2024-04-19T02:03:06.961 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-04-19T02:03:06.961 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_feed_pullee": "-1", 2024-04-19T02:03:06.961 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-04-19T02:03:06.961 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-04-19T02:03:06.961 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-04-19T02:03:06.961 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_misdirected_ops": "false", 2024-04-19T02:03:06.961 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_no_acting_change": "false", 2024-04-19T02:03:06.961 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_no_purge_strays": "false", 2024-04-19T02:03:06.961 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_op_order": "false", 2024-04-19T02:03:06.961 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_pg_log_writeout": "false", 2024-04-19T02:03:06.961 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_pretend_recovery_active": "false", 2024-04-19T02:03:06.962 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-04-19T02:03:06.962 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-04-19T02:03:06.962 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_shutdown": "false", 2024-04-19T02:03:06.962 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-04-19T02:03:06.962 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-04-19T02:03:06.962 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_trim_objects": "false", 2024-04-19T02:03:06.962 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_verify_cached_snaps": "false", 2024-04-19T02:03:06.962 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_verify_missing_on_start": "false", 2024-04-19T02:03:06.962 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_verify_snaps": "false", 2024-04-19T02:03:06.962 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-04-19T02:03:06.962 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-04-19T02:03:06.962 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_deep_scrub_keys": "1024", 2024-04-19T02:03:06.962 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-04-19T02:03:06.962 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-04-19T02:03:06.963 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-04-19T02:03:06.963 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_deep_scrub_stride": "524288", 2024-04-19T02:03:06.963 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-04-19T02:03:06.963 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_default_data_pool_replay_window": "5", 2024-04-19T02:03:06.963 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_default_notify_timeout": "30", 2024-04-19T02:03:06.963 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_delete_sleep": "0.000000", 2024-04-19T02:03:06.963 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-04-19T02:03:06.963 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-04-19T02:03:06.963 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-04-19T02:03:06.963 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_discard_disconnected_ops": "true", 2024-04-19T02:03:06.963 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_enable_op_tracker": "true", 2024-04-19T02:03:06.963 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-04-19T02:03:06.963 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_extblkdev_plugins": "vdo", 2024-04-19T02:03:06.963 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-04-19T02:03:06.963 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-04-19T02:03:06.963 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_fast_info": "true", 2024-04-19T02:03:06.963 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_fast_shutdown": "true", 2024-04-19T02:03:06.964 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-04-19T02:03:06.964 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_fast_shutdown_timeout": "15", 2024-04-19T02:03:06.964 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-04-19T02:03:06.964 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-04-19T02:03:06.964 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-04-19T02:03:06.964 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_function_tracing": "false", 2024-04-19T02:03:06.964 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_heartbeat_grace": "20", 2024-04-19T02:03:06.964 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_heartbeat_interval": "6", 2024-04-19T02:03:06.964 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-04-19T02:03:06.964 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_heartbeat_min_peers": "10", 2024-04-19T02:03:06.964 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_heartbeat_min_size": "2000", 2024-04-19T02:03:06.964 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_heartbeat_stale": "600", 2024-04-19T02:03:06.964 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-04-19T02:03:06.964 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_hit_set_max_size": "100000", 2024-04-19T02:03:06.964 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_hit_set_min_size": "1000", 2024-04-19T02:03:06.965 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-04-19T02:03:06.965 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-04-19T02:03:06.965 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-04-19T02:03:06.965 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-04-19T02:03:06.965 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-04-19T02:03:06.965 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_journal_flush_on_shutdown": "true", 2024-04-19T02:03:06.965 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_journal_size": "5120", 2024-04-19T02:03:06.965 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_kill_backfill_at": "0", 2024-04-19T02:03:06.965 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_loop_before_reset_tphandle": "64", 2024-04-19T02:03:06.965 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_map_cache_size": "50", 2024-04-19T02:03:06.965 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_map_dedup": "true", 2024-04-19T02:03:06.965 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_map_message_max": "40", 2024-04-19T02:03:06.965 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_map_message_max_bytes": "10485760", 2024-04-19T02:03:06.965 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_map_share_max_epochs": "40", 2024-04-19T02:03:06.965 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_attr_name_len": "100", 2024-04-19T02:03:06.965 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_attr_size": "0", 2024-04-19T02:03:06.965 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_backfills": "1", 2024-04-19T02:03:06.966 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_markdown_count": "5", 2024-04-19T02:03:06.966 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_markdown_period": "600", 2024-04-19T02:03:06.966 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_object_name_len": "2048", 2024-04-19T02:03:06.966 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_object_namespace_len": "256", 2024-04-19T02:03:06.966 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_object_size": "134217728", 2024-04-19T02:03:06.966 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-04-19T02:03:06.966 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_omap_entries_per_request": "1024", 2024-04-19T02:03:06.966 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_pg_blocked_by": "16", 2024-04-19T02:03:06.966 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_pg_log_entries": "10000", 2024-04-19T02:03:06.966 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-04-19T02:03:06.966 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_pgls": "1024", 2024-04-19T02:03:06.966 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_push_cost": "8388608", 2024-04-19T02:03:06.966 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_push_objects": "10", 2024-04-19T02:03:06.966 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_scrubs": "3", 2024-04-19T02:03:06.966 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-04-19T02:03:06.967 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_trimming_pgs": "2", 2024-04-19T02:03:06.967 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_write_op_reply_len": "64", 2024-04-19T02:03:06.967 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_max_write_size": "90", 2024-04-19T02:03:06.967 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-04-19T02:03:06.967 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-04-19T02:03:06.967 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-04-19T02:03:06.967 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-04-19T02:03:06.967 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-04-19T02:03:06.967 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-04-19T02:03:06.967 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-04-19T02:03:06.967 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_override_recovery_settings": "false", 2024-04-19T02:03:06.967 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_profile": "balanced", 2024-04-19T02:03:06.967 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-04-19T02:03:06.967 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-04-19T02:03:06.967 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-04-19T02:03:06.968 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-04-19T02:03:06.968 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-04-19T02:03:06.968 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-04-19T02:03:06.968 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-04-19T02:03:06.968 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-04-19T02:03:06.968 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-04-19T02:03:06.968 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-04-19T02:03:06.968 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mclock_skip_benchmark": "false", 2024-04-19T02:03:06.968 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_memory_base": "805306368", 2024-04-19T02:03:06.968 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_memory_cache_min": "134217728", 2024-04-19T02:03:06.968 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-04-19T02:03:06.968 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-04-19T02:03:06.968 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_memory_target": "4294967296", 2024-04-19T02:03:06.968 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_memory_target_autotune": "false", 2024-04-19T02:03:06.968 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-04-19T02:03:06.969 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_min_pg_log_entries": "250", 2024-04-19T02:03:06.969 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_min_recovery_priority": "0", 2024-04-19T02:03:06.969 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mon_heartbeat_interval": "30", 2024-04-19T02:03:06.969 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-04-19T02:03:06.969 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mon_report_interval": "5", 2024-04-19T02:03:06.969 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mon_report_max_in_flight": "2", 2024-04-19T02:03:06.969 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-04-19T02:03:06.969 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_num_cache_shards": "32", 2024-04-19T02:03:06.969 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_num_op_tracker_shard": "32", 2024-04-19T02:03:06.969 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_numa_auto_affinity": "true", 2024-04-19T02:03:06.969 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_numa_node": "-1", 2024-04-19T02:03:06.969 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_numa_prefer_iface": "true", 2024-04-19T02:03:06.969 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_object_clean_region_max_num_intervals": "1000", 2024-04-19T02:03:06.969 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_objecter_finishers": "1", 2024-04-19T02:03:06.970 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_objectstore": "bluestore", 2024-04-19T02:03:06.970 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_objectstore_fuse": "false", 2024-04-19T02:03:06.970 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_objectstore_tracing": "false", 2024-04-19T02:03:06.970 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_complaint_time": "30.000000", 2024-04-19T02:03:06.970 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_history_duration": "600", 2024-04-19T02:03:06.970 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_history_size": "20", 2024-04-19T02:03:06.970 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_history_slow_op_size": "20", 2024-04-19T02:03:06.970 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-04-19T02:03:06.970 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_log_threshold": "5", 2024-04-19T02:03:06.970 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_num_shards": "0", 2024-04-19T02:03:06.970 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_num_shards_hdd": "5", 2024-04-19T02:03:06.970 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_num_shards_ssd": "8", 2024-04-19T02:03:06.970 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_num_threads_per_shard": "0", 2024-04-19T02:03:06.970 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-04-19T02:03:06.970 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-04-19T02:03:06.971 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-04-19T02:03:06.971 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_pq_min_cost": "65536", 2024-04-19T02:03:06.971 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_queue": "mclock_scheduler", 2024-04-19T02:03:06.971 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_queue_cut_off": "high", 2024-04-19T02:03:06.971 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_thread_suicide_timeout": "150", 2024-04-19T02:03:06.971 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_op_thread_timeout": "15", 2024-04-19T02:03:06.971 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_open_classes_on_start": "true", 2024-04-19T02:03:06.971 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_os_flags": "0", 2024-04-19T02:03:06.971 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_peering_op_priority": "255", 2024-04-19T02:03:06.971 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_delete_cost": "1048576", 2024-04-19T02:03:06.971 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_delete_priority": "5", 2024-04-19T02:03:06.971 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-04-19T02:03:06.971 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-04-19T02:03:06.971 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_log_dups_tracked": "3000", 2024-04-19T02:03:06.971 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_log_trim_max": "10000", 2024-04-19T02:03:06.971 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_log_trim_min": "100", 2024-04-19T02:03:06.972 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-04-19T02:03:06.972 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_object_context_cache_count": "64", 2024-04-19T02:03:06.972 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-04-19T02:03:06.972 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-04-19T02:03:06.972 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-04-19T02:03:06.972 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-04-19T02:03:06.972 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-04-19T02:03:06.972 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-04-19T02:03:06.972 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-04-19T02:03:06.972 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-04-19T02:03:06.972 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_crimson": "false", 2024-04-19T02:03:06.972 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_crush_rule": "-1", 2024-04-19T02:03:06.972 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_ec_fast_read": "false", 2024-04-19T02:03:06.972 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-04-19T02:03:06.972 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_flag_bulk": "false", 2024-04-19T02:03:06.973 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-04-19T02:03:06.973 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_flag_nodelete": "false", 2024-04-19T02:03:06.973 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-04-19T02:03:06.973 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-04-19T02:03:06.973 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_flags": "0", 2024-04-19T02:03:06.973 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-04-19T02:03:06.973 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_min_size": "0", 2024-04-19T02:03:06.973 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-04-19T02:03:06.973 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_pg_num": "32", 2024-04-19T02:03:06.973 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_pgp_num": "0", 2024-04-19T02:03:06.973 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-04-19T02:03:06.973 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_read_ratio": "70", 2024-04-19T02:03:06.973 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_size": "2", 2024-04-19T02:03:06.973 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_default_type": "replicated", 2024-04-19T02:03:06.973 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-04-19T02:03:06.974 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_pool_use_gmt_hitset": "true", 2024-04-19T02:03:06.974 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_push_per_object_cost": "1000", 2024-04-19T02:03:06.974 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_read_ec_check_for_errors": "false", 2024-04-19T02:03:06.974 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recover_clone_overlap": "true", 2024-04-19T02:03:06.974 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recover_clone_overlap_limit": "10", 2024-04-19T02:03:06.974 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_cost": "20971520", 2024-04-19T02:03:06.974 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_delay_start": "0.000000", 2024-04-19T02:03:06.974 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_max_active": "0", 2024-04-19T02:03:06.974 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_max_active_hdd": "3", 2024-04-19T02:03:06.974 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_max_active_ssd": "10", 2024-04-19T02:03:06.974 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_max_chunk": "8388608", 2024-04-19T02:03:06.974 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-04-19T02:03:06.974 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_max_single_start": "1", 2024-04-19T02:03:06.974 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_op_priority": "3", 2024-04-19T02:03:06.974 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_op_warn_multiple": "16", 2024-04-19T02:03:06.975 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_priority": "5", 2024-04-19T02:03:06.975 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_retry_interval": "30.000000", 2024-04-19T02:03:06.975 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_sleep": "0.000000", 2024-04-19T02:03:06.975 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-04-19T02:03:06.975 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-04-19T02:03:06.975 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-04-19T02:03:06.975 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_repair_during_recovery": "false", 2024-04-19T02:03:06.975 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_requested_scrub_priority": "120", 2024-04-19T02:03:06.975 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-04-19T02:03:06.975 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_rollback_to_cluster_snap": "", 2024-04-19T02:03:06.975 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_auto_repair": "false", 2024-04-19T02:03:06.975 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-04-19T02:03:06.975 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-04-19T02:03:06.975 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_begin_hour": "0", 2024-04-19T02:03:06.976 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_begin_week_day": "0", 2024-04-19T02:03:06.976 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_chunk_max": "25", 2024-04-19T02:03:06.976 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_chunk_min": "5", 2024-04-19T02:03:06.976 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_cost": "52428800", 2024-04-19T02:03:06.976 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_disable_reservation_queuing": "true", 2024-04-19T02:03:06.976 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_during_recovery": "false", 2024-04-19T02:03:06.976 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_end_hour": "0", 2024-04-19T02:03:06.976 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_end_week_day": "0", 2024-04-19T02:03:06.976 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_event_cost": "4096", 2024-04-19T02:03:06.976 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-04-19T02:03:06.976 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-04-19T02:03:06.976 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_invalid_stats": "true", 2024-04-19T02:03:06.976 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_load_threshold": "0.500000", 2024-04-19T02:03:06.976 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_max_interval": "604800.000000", 2024-04-19T02:03:06.976 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_max_preemptions": "5", 2024-04-19T02:03:06.977 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_min_interval": "86400.000000", 2024-04-19T02:03:06.977 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_priority": "5", 2024-04-19T02:03:06.977 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_reservation_timeout": "300000", 2024-04-19T02:03:06.977 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_sleep": "0.000000", 2024-04-19T02:03:06.977 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-04-19T02:03:06.977 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-04-19T02:03:06.977 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-04-19T02:03:06.977 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_shutdown_pgref_assert": "false", 2024-04-19T02:03:06.977 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_skip_data_digest": "false", 2024-04-19T02:03:06.977 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_smart_report_timeout": "5", 2024-04-19T02:03:06.977 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_snap_trim_cost": "1048576", 2024-04-19T02:03:06.977 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_snap_trim_priority": "5", 2024-04-19T02:03:06.977 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_snap_trim_sleep": "0.000000", 2024-04-19T02:03:06.977 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-04-19T02:03:06.977 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-04-19T02:03:06.978 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-04-19T02:03:06.978 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-04-19T02:03:06.978 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_stats_update_period_scrubbing": "15", 2024-04-19T02:03:06.978 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-04-19T02:03:06.978 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_target_transaction_size": "30", 2024-04-19T02:03:06.978 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-04-19T02:03:06.978 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-04-19T02:03:06.978 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-04-19T02:03:06.978 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-04-19T02:03:06.978 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-04-19T02:03:06.978 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-04-19T02:03:06.978 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-04-19T02:03:06.978 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tier_default_cache_mode": "writeback", 2024-04-19T02:03:06.978 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-04-19T02:03:06.978 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-04-19T02:03:06.979 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_tracing": "false", 2024-04-19T02:03:06.979 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_use_stale_snap": "false", 2024-04-19T02:03:06.979 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-04-19T02:03:06.979 INFO:teuthology.orchestra.run.smithi059.stdout: "osdc_blkin_trace_all": "false", 2024-04-19T02:03:06.979 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_kill_at": "0", 2024-04-19T02:03:06.979 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_max_join_drift": "10", 2024-04-19T02:03:06.979 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_min": "500", 2024-04-19T02:03:06.979 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_min_wait": "0.050000", 2024-04-19T02:03:06.979 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_propose_interval": "1.000000", 2024-04-19T02:03:06.979 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_service_trim_max": "500", 2024-04-19T02:03:06.979 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_service_trim_max_multiplier": "20", 2024-04-19T02:03:06.979 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_service_trim_min": "250", 2024-04-19T02:03:06.979 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_stash_full_interval": "25", 2024-04-19T02:03:06.979 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_trim_max": "500", 2024-04-19T02:03:06.979 INFO:teuthology.orchestra.run.smithi059.stdout: "paxos_trim_min": "250", 2024-04-19T02:03:06.980 INFO:teuthology.orchestra.run.smithi059.stdout: "perf": "true", 2024-04-19T02:03:06.980 INFO:teuthology.orchestra.run.smithi059.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-04-19T02:03:06.980 INFO:teuthology.orchestra.run.smithi059.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-04-19T02:03:06.980 INFO:teuthology.orchestra.run.smithi059.stdout: "plugin_dir": "/usr/lib/ceph", 2024-04-19T02:03:06.980 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addr": "-", 2024-04-19T02:03:06.980 INFO:teuthology.orchestra.run.smithi059.stdout: "public_addrv": "", 2024-04-19T02:03:06.980 INFO:teuthology.orchestra.run.smithi059.stdout: "public_bind_addr": "-", 2024-04-19T02:03:06.980 INFO:teuthology.orchestra.run.smithi059.stdout: "public_network": "", 2024-04-19T02:03:06.980 INFO:teuthology.orchestra.run.smithi059.stdout: "public_network_interface": "", 2024-04-19T02:03:06.980 INFO:teuthology.orchestra.run.smithi059.stdout: "qat_compressor_enabled": "false", 2024-04-19T02:03:06.980 INFO:teuthology.orchestra.run.smithi059.stdout: "qat_compressor_session_max_number": "256", 2024-04-19T02:03:06.980 INFO:teuthology.orchestra.run.smithi059.stdout: "rados_mon_op_timeout": "0", 2024-04-19T02:03:06.980 INFO:teuthology.orchestra.run.smithi059.stdout: "rados_osd_op_timeout": "0", 2024-04-19T02:03:06.980 INFO:teuthology.orchestra.run.smithi059.stdout: "rados_tracing": "false", 2024-04-19T02:03:06.980 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_atime_update_interval": "60", 2024-04-19T02:03:06.981 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-04-19T02:03:06.981 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_balance_parent_reads": "false", 2024-04-19T02:03:06.981 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_balance_snap_reads": "false", 2024-04-19T02:03:06.981 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_blkin_trace_all": "false", 2024-04-19T02:03:06.981 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_blocklist_expire_seconds": "0", 2024-04-19T02:03:06.981 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_blocklist_on_break_lock": "true", 2024-04-19T02:03:06.981 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_cache": "true", 2024-04-19T02:03:06.981 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_cache_block_writes_upfront": "false", 2024-04-19T02:03:06.981 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_cache_max_dirty": "25165824", 2024-04-19T02:03:06.981 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-04-19T02:03:06.981 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_cache_max_dirty_object": "0", 2024-04-19T02:03:06.981 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_cache_policy": "writearound", 2024-04-19T02:03:06.981 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_cache_size": "33554432", 2024-04-19T02:03:06.981 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_cache_target_dirty": "16777216", 2024-04-19T02:03:06.981 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-04-19T02:03:06.982 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_clone_copy_on_read": "false", 2024-04-19T02:03:06.982 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_compression_hint": "none", 2024-04-19T02:03:06.982 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_concurrent_management_ops": "10", 2024-04-19T02:03:06.982 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-04-19T02:03:06.982 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_default_clone_format": "auto", 2024-04-19T02:03:06.982 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_default_data_pool": "", 2024-04-19T02:03:06.982 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_default_features": "61", 2024-04-19T02:03:06.982 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_default_format": "2", 2024-04-19T02:03:06.982 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_default_map_options": "", 2024-04-19T02:03:06.982 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_default_order": "22", 2024-04-19T02:03:06.982 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_default_pool": "rbd", 2024-04-19T02:03:06.982 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-04-19T02:03:06.982 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_default_stripe_count": "0", 2024-04-19T02:03:06.982 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_default_stripe_unit": "0", 2024-04-19T02:03:06.982 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_disable_zero_copy_writes": "true", 2024-04-19T02:03:06.983 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_discard_granularity_bytes": "65536", 2024-04-19T02:03:06.983 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-04-19T02:03:06.983 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_enable_alloc_hint": "true", 2024-04-19T02:03:06.983 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-04-19T02:03:06.983 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_io_scheduler": "simple", 2024-04-19T02:03:06.983 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-04-19T02:03:06.983 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_commit_age": "5.000000", 2024-04-19T02:03:06.983 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-04-19T02:03:06.983 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-04-19T02:03:06.983 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-04-19T02:03:06.983 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-04-19T02:03:06.983 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_object_flush_interval": "0", 2024-04-19T02:03:06.983 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-04-19T02:03:06.983 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-04-19T02:03:06.984 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_order": "24", 2024-04-19T02:03:06.984 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_pool": "", 2024-04-19T02:03:06.984 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_journal_splay_width": "4", 2024-04-19T02:03:06.984 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_localize_parent_reads": "false", 2024-04-19T02:03:06.984 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_localize_snap_reads": "false", 2024-04-19T02:03:06.984 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-04-19T02:03:06.984 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-04-19T02:03:06.984 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-04-19T02:03:06.984 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-04-19T02:03:06.984 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-04-19T02:03:06.984 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-04-19T02:03:06.984 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_image_policy_type": "simple", 2024-04-19T02:03:06.984 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-04-19T02:03:06.984 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-04-19T02:03:06.984 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-04-19T02:03:06.985 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-04-19T02:03:06.985 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-04-19T02:03:06.985 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-04-19T02:03:06.985 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-04-19T02:03:06.985 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_memory_autotune": "true", 2024-04-19T02:03:06.985 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_memory_base": "805306368", 2024-04-19T02:03:06.985 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-04-19T02:03:06.985 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-04-19T02:03:06.985 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-04-19T02:03:06.985 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-04-19T02:03:06.985 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_memory_target": "4294967296", 2024-04-19T02:03:06.985 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-04-19T02:03:06.985 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-04-19T02:03:06.985 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-04-19T02:03:06.985 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirroring_delete_delay": "0", 2024-04-19T02:03:06.986 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-04-19T02:03:06.986 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirroring_replay_delay": "0", 2024-04-19T02:03:06.986 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-04-19T02:03:06.986 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-04-19T02:03:06.986 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_move_to_trash_on_remove": "false", 2024-04-19T02:03:06.986 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-04-19T02:03:06.986 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_mtime_update_interval": "60", 2024-04-19T02:03:06.986 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_non_blocking_aio": "true", 2024-04-19T02:03:06.986 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_op_thread_timeout": "60", 2024-04-19T02:03:06.986 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_op_threads": "1", 2024-04-19T02:03:06.986 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_parent_cache_enabled": "false", 2024-04-19T02:03:06.986 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_persistent_cache_mode": "disabled", 2024-04-19T02:03:06.986 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_persistent_cache_path": "/tmp", 2024-04-19T02:03:06.986 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_persistent_cache_size": "1073741824", 2024-04-19T02:03:06.986 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_plugins": "", 2024-04-19T02:03:06.987 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_bps_burst": "0", 2024-04-19T02:03:06.987 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-04-19T02:03:06.987 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_bps_limit": "0", 2024-04-19T02:03:06.987 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_exclude_ops": "0", 2024-04-19T02:03:06.987 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_iops_burst": "0", 2024-04-19T02:03:06.987 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-04-19T02:03:06.987 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_iops_limit": "0", 2024-04-19T02:03:06.987 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_read_bps_burst": "0", 2024-04-19T02:03:06.987 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-04-19T02:03:06.987 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_read_bps_limit": "0", 2024-04-19T02:03:06.987 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_read_iops_burst": "0", 2024-04-19T02:03:06.987 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-04-19T02:03:06.987 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_read_iops_limit": "0", 2024-04-19T02:03:06.987 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_schedule_tick_min": "50", 2024-04-19T02:03:06.987 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_write_bps_burst": "0", 2024-04-19T02:03:06.988 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-04-19T02:03:06.988 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_write_bps_limit": "0", 2024-04-19T02:03:06.988 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_write_iops_burst": "0", 2024-04-19T02:03:06.988 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-04-19T02:03:06.988 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_qos_write_iops_limit": "0", 2024-04-19T02:03:06.988 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_quiesce_notification_attempts": "10", 2024-04-19T02:03:06.988 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_read_from_replica_policy": "default", 2024-04-19T02:03:06.988 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-04-19T02:03:06.988 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_readahead_max_bytes": "524288", 2024-04-19T02:03:06.988 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_readahead_trigger_requests": "10", 2024-04-19T02:03:06.988 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_request_timed_out_seconds": "30", 2024-04-19T02:03:06.988 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_skip_partial_discard": "true", 2024-04-19T02:03:06.988 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-04-19T02:03:06.988 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_tracing": "false", 2024-04-19T02:03:06.988 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_validate_names": "true", 2024-04-19T02:03:06.988 INFO:teuthology.orchestra.run.smithi059.stdout: "rbd_validate_pool": "true", 2024-04-19T02:03:06.989 INFO:teuthology.orchestra.run.smithi059.stdout: "restapi_base_url": "", 2024-04-19T02:03:06.989 INFO:teuthology.orchestra.run.smithi059.stdout: "restapi_log_level": "", 2024-04-19T02:03:06.989 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_acl_grants_max_num": "100", 2024-04-19T02:03:06.989 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_admin_entry": "admin", 2024-04-19T02:03:06.989 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-04-19T02:03:06.989 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_backend_store": "rados", 2024-04-19T02:03:06.989 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_barbican_url": "", 2024-04-19T02:03:06.989 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_beast_enable_async": "true", 2024-04-19T02:03:06.989 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_bucket_counters_cache": "false", 2024-04-19T02:03:06.989 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-04-19T02:03:06.989 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-04-19T02:03:06.989 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-04-19T02:03:06.989 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_bucket_index_max_aio": "128", 2024-04-19T02:03:06.989 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-04-19T02:03:06.990 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-04-19T02:03:06.990 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_bucket_quota_ttl": "600", 2024-04-19T02:03:06.990 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-04-19T02:03:06.990 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_cache_enabled": "true", 2024-04-19T02:03:06.990 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_cache_expiry_interval": "900", 2024-04-19T02:03:06.990 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_cache_lru_size": "10000", 2024-04-19T02:03:06.990 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_config_store": "rados", 2024-04-19T02:03:06.990 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_content_length_compat": "false", 2024-04-19T02:03:06.990 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_copy_obj_progress": "true", 2024-04-19T02:03:06.990 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-04-19T02:03:06.990 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_cors_rules_max_num": "100", 2024-04-19T02:03:06.990 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_cross_domain_policy": "", 2024-04-19T02:03:06.990 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_default_encryption_key": "", 2024-04-19T02:03:06.990 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_kmip_addr": "", 2024-04-19T02:03:06.990 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_kmip_ca_path": "", 2024-04-19T02:03:06.991 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_kmip_client_cert": "", 2024-04-19T02:03:06.991 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_kmip_client_key": "", 2024-04-19T02:03:06.991 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-04-19T02:03:06.991 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_kmip_password": "", 2024-04-19T02:03:06.991 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-04-19T02:03:06.991 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_kmip_username": "", 2024-04-19T02:03:06.991 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_require_ssl": "true", 2024-04-19T02:03:06.991 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-04-19T02:03:06.991 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-04-19T02:03:06.991 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-04-19T02:03:06.991 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-04-19T02:03:06.991 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-04-19T02:03:06.991 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-04-19T02:03:06.991 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-04-19T02:03:06.992 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-04-19T02:03:06.992 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-04-19T02:03:06.992 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-04-19T02:03:06.992 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-04-19T02:03:06.992 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-04-19T02:03:06.992 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-04-19T02:03:06.992 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-04-19T02:03:06.992 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_suppress_logs": "true", 2024-04-19T02:03:06.992 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_vault_addr": "", 2024-04-19T02:03:06.992 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_vault_auth": "token", 2024-04-19T02:03:06.992 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_vault_namespace": "", 2024-04-19T02:03:06.992 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_vault_prefix": "", 2024-04-19T02:03:06.992 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-04-19T02:03:06.992 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-04-19T02:03:06.992 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-04-19T02:03:06.993 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-04-19T02:03:06.993 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_vault_token_file": "", 2024-04-19T02:03:06.993 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-04-19T02:03:06.993 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_curl_buffersize": "524288", 2024-04-19T02:03:06.993 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_curl_low_speed_limit": "1024", 2024-04-19T02:03:06.993 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_curl_low_speed_time": "300", 2024-04-19T02:03:06.993 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_curl_tcp_keepalive": "0", 2024-04-19T02:03:06.993 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-04-19T02:03:06.993 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-04-19T02:03:06.993 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-04-19T02:03:06.993 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-04-19T02:03:06.993 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-04-19T02:03:06.993 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_d3n_l1_fadvise": "4", 2024-04-19T02:03:06.993 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-04-19T02:03:06.993 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-04-19T02:03:06.994 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-04-19T02:03:06.994 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_d4n_host": "127.0.0.1", 2024-04-19T02:03:06.994 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_d4n_port": "6379", 2024-04-19T02:03:06.994 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-04-19T02:03:06.994 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_data_log_changes_size": "1000", 2024-04-19T02:03:06.994 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_data_log_num_shards": "128", 2024-04-19T02:03:06.994 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-04-19T02:03:06.994 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_data_log_window": "30", 2024-04-19T02:03:06.994 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_data_notify_interval_msec": "0", 2024-04-19T02:03:06.994 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_data_sync_poll_interval": "20", 2024-04-19T02:03:06.994 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_data_sync_spawn_window": "20", 2024-04-19T02:03:06.994 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-04-19T02:03:06.994 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-04-19T02:03:06.994 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-04-19T02:03:06.995 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_default_data_log_backing": "fifo", 2024-04-19T02:03:06.995 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-04-19T02:03:06.995 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_default_region_info_oid": "default.region", 2024-04-19T02:03:06.995 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-04-19T02:03:06.995 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-04-19T02:03:06.995 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_defer_to_bucket_acls": "", 2024-04-19T02:03:06.995 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-04-19T02:03:06.995 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-04-19T02:03:06.995 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-04-19T02:03:06.995 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-04-19T02:03:06.995 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-04-19T02:03:06.995 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-04-19T02:03:06.995 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-04-19T02:03:06.995 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-04-19T02:03:06.995 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_data_res": "500.000000", 2024-04-19T02:03:06.996 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-04-19T02:03:06.996 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-04-19T02:03:06.996 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-04-19T02:03:06.996 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-04-19T02:03:06.996 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dns_name": "", 2024-04-19T02:03:06.996 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dns_s3website_name": "", 2024-04-19T02:03:06.996 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_dynamic_resharding": "true", 2024-04-19T02:03:06.996 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-04-19T02:03:06.996 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_enable_gc_threads": "true", 2024-04-19T02:03:06.996 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_enable_lc_threads": "true", 2024-04-19T02:03:06.996 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_enable_ops_log": "false", 2024-04-19T02:03:06.996 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_enable_quota_threads": "true", 2024-04-19T02:03:06.996 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_enable_static_website": "false", 2024-04-19T02:03:06.996 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_enable_usage_log": "false", 2024-04-19T02:03:06.996 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_enforce_swift_acls": "true", 2024-04-19T02:03:06.997 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_exit_timeout_secs": "120", 2024-04-19T02:03:06.997 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_expose_bucket": "false", 2024-04-19T02:03:06.997 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_extended_http_attrs": "", 2024-04-19T02:03:06.997 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_filter": "none", 2024-04-19T02:03:06.997 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-04-19T02:03:06.997 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_frontends": "beast port=7480", 2024-04-19T02:03:06.997 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_gc_max_concurrent_io": "10", 2024-04-19T02:03:06.997 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_gc_max_deferred": "50", 2024-04-19T02:03:06.997 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-04-19T02:03:06.997 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_gc_max_objs": "32", 2024-04-19T02:03:06.997 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_gc_max_queue_size": "134213632", 2024-04-19T02:03:06.997 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_gc_max_trim_chunk": "16", 2024-04-19T02:03:06.997 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_gc_obj_min_wait": "7200", 2024-04-19T02:03:06.997 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_gc_processor_max_time": "3600", 2024-04-19T02:03:06.997 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_gc_processor_period": "3600", 2024-04-19T02:03:06.997 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-04-19T02:03:06.998 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_get_obj_window_size": "16777216", 2024-04-19T02:03:06.998 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_healthcheck_disabling_path": "", 2024-04-19T02:03:06.998 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ignore_get_invalid_range": "false", 2024-04-19T02:03:06.998 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_init_timeout": "300", 2024-04-19T02:03:06.998 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-04-19T02:03:06.998 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-04-19T02:03:06.998 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_kafka_connection_idle": "300", 2024-04-19T02:03:06.998 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_kafka_message_timeout": "5000", 2024-04-19T02:03:06.998 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_kafka_sleep_timeout": "10", 2024-04-19T02:03:06.998 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-04-19T02:03:06.998 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-04-19T02:03:06.998 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-04-19T02:03:06.998 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_admin_domain": "", 2024-04-19T02:03:06.998 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_admin_password": "", 2024-04-19T02:03:06.999 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_admin_password_path": "", 2024-04-19T02:03:06.999 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_admin_project": "", 2024-04-19T02:03:06.999 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_admin_tenant": "", 2024-04-19T02:03:06.999 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_admin_token": "", 2024-04-19T02:03:06.999 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_admin_token_path": "", 2024-04-19T02:03:06.999 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_admin_user": "", 2024-04-19T02:03:06.999 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_api_version": "2", 2024-04-19T02:03:06.999 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_barbican_domain": "", 2024-04-19T02:03:06.999 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_barbican_password": "", 2024-04-19T02:03:06.999 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_barbican_project": "", 2024-04-19T02:03:06.999 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_barbican_tenant": "", 2024-04-19T02:03:06.999 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_barbican_user": "", 2024-04-19T02:03:06.999 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-04-19T02:03:06.999 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_implicit_tenants": "false", 2024-04-19T02:03:06.999 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-04-19T02:03:07.000 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_service_token_enabled": "false", 2024-04-19T02:03:07.000 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_token_cache_size": "10000", 2024-04-19T02:03:07.000 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_url": "", 2024-04-19T02:03:07.000 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_keystone_verify_ssl": "true", 2024-04-19T02:03:07.000 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_lc_debug_interval": "-1", 2024-04-19T02:03:07.000 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_lc_lock_max_time": "90", 2024-04-19T02:03:07.000 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_lc_max_objs": "32", 2024-04-19T02:03:07.000 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_lc_max_rules": "1000", 2024-04-19T02:03:07.000 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_lc_max_worker": "3", 2024-04-19T02:03:07.000 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_lc_max_wp_worker": "3", 2024-04-19T02:03:07.000 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_lc_thread_delay": "0", 2024-04-19T02:03:07.000 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-04-19T02:03:07.000 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ldap_dnattr": "uid", 2024-04-19T02:03:07.000 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-04-19T02:03:07.001 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ldap_searchfilter": "", 2024-04-19T02:03:07.001 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-04-19T02:03:07.001 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ldap_uri": "", 2024-04-19T02:03:07.001 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-04-19T02:03:07.001 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-04-19T02:03:07.001 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-04-19T02:03:07.001 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_log_http_headers": "", 2024-04-19T02:03:07.001 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_log_nonexistent_bucket": "false", 2024-04-19T02:03:07.001 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-04-19T02:03:07.001 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_log_object_name_utc": "false", 2024-04-19T02:03:07.001 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-04-19T02:03:07.001 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-04-19T02:03:07.001 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_attr_name_len": "0", 2024-04-19T02:03:07.001 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_attr_size": "0", 2024-04-19T02:03:07.002 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_attrs_num_in_req": "0", 2024-04-19T02:03:07.002 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_chunk_size": "4194304", 2024-04-19T02:03:07.002 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_concurrent_requests": "1024", 2024-04-19T02:03:07.002 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-04-19T02:03:07.002 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_dynamic_shards": "1999", 2024-04-19T02:03:07.002 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_listing_results": "1000", 2024-04-19T02:03:07.002 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_notify_retries": "10", 2024-04-19T02:03:07.002 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_objs_per_shard": "100000", 2024-04-19T02:03:07.002 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_put_param_size": "1048576", 2024-04-19T02:03:07.002 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_put_size": "5368709120", 2024-04-19T02:03:07.002 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_max_slo_entries": "1000", 2024-04-19T02:03:07.002 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_md_log_max_shards": "64", 2024-04-19T02:03:07.002 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_md_notify_interval_msec": "200", 2024-04-19T02:03:07.002 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_meta_sync_poll_interval": "20", 2024-04-19T02:03:07.003 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_meta_sync_spawn_window": "20", 2024-04-19T02:03:07.003 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-04-19T02:03:07.003 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_mp_lock_max_time": "600", 2024-04-19T02:03:07.003 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-04-19T02:03:07.003 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_multipart_min_part_size": "5242880", 2024-04-19T02:03:07.003 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-04-19T02:03:07.003 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-04-19T02:03:07.003 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_fhcache_size": "2017", 2024-04-19T02:03:07.003 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-04-19T02:03:07.003 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-04-19T02:03:07.003 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_lru_lanes": "5", 2024-04-19T02:03:07.003 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_max_gc": "300", 2024-04-19T02:03:07.003 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-04-19T02:03:07.003 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_run_gc_threads": "false", 2024-04-19T02:03:07.003 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_run_lc_threads": "false", 2024-04-19T02:03:07.003 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_run_quota_threads": "false", 2024-04-19T02:03:07.004 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_run_sync_thread": "false", 2024-04-19T02:03:07.004 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-04-19T02:03:07.004 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-04-19T02:03:07.004 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_num_async_rados_threads": "32", 2024-04-19T02:03:07.004 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_num_control_oids": "8", 2024-04-19T02:03:07.004 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_numa_node": "-1", 2024-04-19T02:03:07.004 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_obj_stripe_size": "4194304", 2024-04-19T02:03:07.004 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-04-19T02:03:07.004 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_objexp_chunk_size": "100", 2024-04-19T02:03:07.004 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_objexp_gc_interval": "600", 2024-04-19T02:03:07.004 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_objexp_hints_num_shards": "127", 2024-04-19T02:03:07.004 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-04-19T02:03:07.004 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-04-19T02:03:07.004 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_op_thread_timeout": "600", 2024-04-19T02:03:07.005 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_op_tracing": "false", 2024-04-19T02:03:07.005 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_opa_token": "", 2024-04-19T02:03:07.005 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_opa_url": "", 2024-04-19T02:03:07.005 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_opa_verify_ssl": "true", 2024-04-19T02:03:07.005 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-04-19T02:03:07.005 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-04-19T02:03:07.005 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ops_log_rados": "false", 2024-04-19T02:03:07.005 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_ops_log_socket_path": "", 2024-04-19T02:03:07.005 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-04-19T02:03:07.005 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-04-19T02:03:07.005 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-04-19T02:03:07.005 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_period_push_interval": "2.000000", 2024-04-19T02:03:07.005 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_period_push_interval_max": "30.000000", 2024-04-19T02:03:07.005 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_period_root_pool": ".rgw.root", 2024-04-19T02:03:07.006 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-04-19T02:03:07.006 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-04-19T02:03:07.006 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_posix_cache_lanes": "3", 2024-04-19T02:03:07.006 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-04-19T02:03:07.006 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_posix_cache_max_buckets": "100", 2024-04-19T02:03:07.006 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_posix_cache_partitions": "3", 2024-04-19T02:03:07.006 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-04-19T02:03:07.006 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_print_continue": "true", 2024-04-19T02:03:07.006 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_print_prohibited_content_length": "false", 2024-04-19T02:03:07.006 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-04-19T02:03:07.006 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-04-19T02:03:07.006 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-04-19T02:03:07.006 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-04-19T02:03:07.006 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_rados_tracing": "false", 2024-04-19T02:03:07.007 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_realm": "", 2024-04-19T02:03:07.007 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_realm_id": "", 2024-04-19T02:03:07.007 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-04-19T02:03:07.007 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_region": "", 2024-04-19T02:03:07.007 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_region_root_pool": ".rgw.root", 2024-04-19T02:03:07.007 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_relaxed_region_enforcement": "false", 2024-04-19T02:03:07.007 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-04-19T02:03:07.007 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-04-19T02:03:07.007 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_request_uri": "", 2024-04-19T02:03:07.007 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_reshard_batch_size": "64", 2024-04-19T02:03:07.007 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-04-19T02:03:07.007 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_reshard_max_aio": "128", 2024-04-19T02:03:07.007 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_reshard_num_logs": "16", 2024-04-19T02:03:07.007 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_reshard_thread_interval": "600", 2024-04-19T02:03:07.007 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_resolve_cname": "false", 2024-04-19T02:03:07.008 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_rest_getusage_op_compat": "false", 2024-04-19T02:03:07.008 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_run_sync_thread": "true", 2024-04-19T02:03:07.008 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-04-19T02:03:07.008 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-04-19T02:03:07.008 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_s3_auth_use_keystone": "false", 2024-04-19T02:03:07.008 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_s3_auth_use_ldap": "false", 2024-04-19T02:03:07.008 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_s3_auth_use_rados": "true", 2024-04-19T02:03:07.008 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_s3_auth_use_sts": "false", 2024-04-19T02:03:07.008 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-04-19T02:03:07.008 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_s3_success_create_obj_status": "0", 2024-04-19T02:03:07.008 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-04-19T02:03:07.008 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_scheduler_type": "throttler", 2024-04-19T02:03:07.008 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_script_uri": "", 2024-04-19T02:03:07.008 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_service_provider_name": "", 2024-04-19T02:03:07.008 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-04-19T02:03:07.009 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sts_client_id": "", 2024-04-19T02:03:07.009 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sts_client_secret": "", 2024-04-19T02:03:07.009 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sts_entry": "sts", 2024-04-19T02:03:07.009 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sts_key": "sts", 2024-04-19T02:03:07.009 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sts_max_session_duration": "43200", 2024-04-19T02:03:07.009 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sts_min_session_duration": "900", 2024-04-19T02:03:07.009 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sts_token_introspection_url": "", 2024-04-19T02:03:07.009 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_account_in_url": "false", 2024-04-19T02:03:07.009 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_auth_entry": "auth", 2024-04-19T02:03:07.009 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_auth_url": "", 2024-04-19T02:03:07.009 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_custom_header": "", 2024-04-19T02:03:07.009 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_enforce_content_length": "false", 2024-04-19T02:03:07.009 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_need_stats": "true", 2024-04-19T02:03:07.009 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_tenant_name": "", 2024-04-19T02:03:07.009 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_token_expiration": "86400", 2024-04-19T02:03:07.010 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_url": "", 2024-04-19T02:03:07.010 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_url_prefix": "swift", 2024-04-19T02:03:07.010 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_swift_versioning_enabled": "false", 2024-04-19T02:03:07.010 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-04-19T02:03:07.010 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-04-19T02:03:07.010 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_lease_period": "120", 2024-04-19T02:03:07.010 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-04-19T02:03:07.010 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_log_trim_interval": "1200", 2024-04-19T02:03:07.010 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-04-19T02:03:07.010 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-04-19T02:03:07.010 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-04-19T02:03:07.010 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_obj_etag_verify": "false", 2024-04-19T02:03:07.010 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_trace_history_size": "4096", 2024-04-19T02:03:07.010 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-04-19T02:03:07.010 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-04-19T02:03:07.011 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_thread_pool_size": "512", 2024-04-19T02:03:07.011 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_topic_persistency_max_retries": "0", 2024-04-19T02:03:07.011 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-04-19T02:03:07.011 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-04-19T02:03:07.011 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_topic_require_publish_policy": "false", 2024-04-19T02:03:07.011 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_torrent_comment": "", 2024-04-19T02:03:07.011 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_torrent_createby": "", 2024-04-19T02:03:07.011 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_torrent_encoding": "", 2024-04-19T02:03:07.011 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_torrent_flag": "false", 2024-04-19T02:03:07.011 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_torrent_max_size": "5368709120", 2024-04-19T02:03:07.011 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_torrent_origin": "", 2024-04-19T02:03:07.011 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_torrent_sha_unit": "524288", 2024-04-19T02:03:07.011 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_torrent_tracker": "", 2024-04-19T02:03:07.011 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_trust_forwarded_https": "false", 2024-04-19T02:03:07.012 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-04-19T02:03:07.012 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_usage_log_tick_interval": "30", 2024-04-19T02:03:07.012 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_usage_max_shards": "32", 2024-04-19T02:03:07.012 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_usage_max_user_shards": "1", 2024-04-19T02:03:07.012 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_use_opa_authz": "false", 2024-04-19T02:03:07.012 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_counters_cache": "false", 2024-04-19T02:03:07.012 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_counters_cache_size": "10000", 2024-04-19T02:03:07.012 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-04-19T02:03:07.012 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_default_quota_max_size": "-1", 2024-04-19T02:03:07.012 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_max_buckets": "1000", 2024-04-19T02:03:07.012 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_policies_max_num": "100", 2024-04-19T02:03:07.012 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-04-19T02:03:07.012 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-04-19T02:03:07.012 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_quota_sync_interval": "86400", 2024-04-19T02:03:07.012 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-04-19T02:03:07.013 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_user_unique_email": "true", 2024-04-19T02:03:07.013 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_verify_ssl": "true", 2024-04-19T02:03:07.013 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_website_routing_rules_max_num": "50", 2024-04-19T02:03:07.013 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_zone": "", 2024-04-19T02:03:07.013 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_zone_id": "", 2024-04-19T02:03:07.013 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-04-19T02:03:07.013 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_zonegroup": "", 2024-04-19T02:03:07.013 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_zonegroup_id": "", 2024-04-19T02:03:07.013 INFO:teuthology.orchestra.run.smithi059.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-04-19T02:03:07.013 INFO:teuthology.orchestra.run.smithi059.stdout: "rgwlc_auto_session_clear": "true", 2024-04-19T02:03:07.013 INFO:teuthology.orchestra.run.smithi059.stdout: "rgwlc_skip_bucket_step": "false", 2024-04-19T02:03:07.013 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_block_size": "4096", 2024-04-19T02:03:07.013 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-04-19T02:03:07.013 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-04-19T02:03:07.013 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-04-19T02:03:07.013 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-04-19T02:03:07.014 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_cache_shard_bits": "4", 2024-04-19T02:03:07.014 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_cache_size": "536870912", 2024-04-19T02:03:07.014 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_cache_type": "binned_lru", 2024-04-19T02:03:07.014 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-04-19T02:03:07.014 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-04-19T02:03:07.014 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-04-19T02:03:07.014 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_collect_compaction_stats": "false", 2024-04-19T02:03:07.014 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_collect_extended_stats": "false", 2024-04-19T02:03:07.014 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_collect_memory_stats": "false", 2024-04-19T02:03:07.014 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-04-19T02:03:07.014 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_index_type": "binary_search", 2024-04-19T02:03:07.014 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_log_to_ceph_log": "true", 2024-04-19T02:03:07.014 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_metadata_block_size": "4096", 2024-04-19T02:03:07.014 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_partition_filters": "false", 2024-04-19T02:03:07.015 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_perf": "false", 2024-04-19T02:03:07.015 INFO:teuthology.orchestra.run.smithi059.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-04-19T02:03:07.015 INFO:teuthology.orchestra.run.smithi059.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-04-19T02:03:07.015 INFO:teuthology.orchestra.run.smithi059.stdout: "rotating_keys_renewal_timeout": "10", 2024-04-19T02:03:07.015 INFO:teuthology.orchestra.run.smithi059.stdout: "run_dir": "/var/run/ceph", 2024-04-19T02:03:07.015 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_block_create": "true", 2024-04-19T02:03:07.015 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_cache_lru_size": "67108864", 2024-04-19T02:03:07.015 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_cbjournal_size": "5368709120", 2024-04-19T02:03:07.015 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_data_delta_based_overwrite": "0", 2024-04-19T02:03:07.015 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_default_max_object_size": "16777216", 2024-04-19T02:03:07.015 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-04-19T02:03:07.015 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_device_size": "53687091200", 2024-04-19T02:03:07.015 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_journal_batch_capacity": "16", 2024-04-19T02:03:07.015 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-04-19T02:03:07.015 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-04-19T02:03:07.015 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_journal_iodepth_limit": "5", 2024-04-19T02:03:07.016 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_main_device_type": "SSD", 2024-04-19T02:03:07.016 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_max_concurrent_transactions": "8", 2024-04-19T02:03:07.016 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-04-19T02:03:07.016 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-04-19T02:03:07.016 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-04-19T02:03:07.016 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-04-19T02:03:07.016 INFO:teuthology.orchestra.run.smithi059.stdout: "seastore_segment_size": "67108864", 2024-04-19T02:03:07.016 INFO:teuthology.orchestra.run.smithi059.stdout: "set_keepcaps": "false", 2024-04-19T02:03:07.016 INFO:teuthology.orchestra.run.smithi059.stdout: "setgroup": "", 2024-04-19T02:03:07.016 INFO:teuthology.orchestra.run.smithi059.stdout: "setuser": "", 2024-04-19T02:03:07.016 INFO:teuthology.orchestra.run.smithi059.stdout: "setuser_match_path": "", 2024-04-19T02:03:07.016 INFO:teuthology.orchestra.run.smithi059.stdout: "target_max_misplaced_ratio": "0.050000", 2024-04-19T02:03:07.016 INFO:teuthology.orchestra.run.smithi059.stdout: "thp": "false", 2024-04-19T02:03:07.016 INFO:teuthology.orchestra.run.smithi059.stdout: "threadpool_default_timeout": "60", 2024-04-19T02:03:07.016 INFO:teuthology.orchestra.run.smithi059.stdout: "threadpool_empty_queue_max_wait": "2", 2024-04-19T02:03:07.017 INFO:teuthology.orchestra.run.smithi059.stdout: "throttler_perf_counter": "true" 2024-04-19T02:03:07.017 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:07.019 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-04-19T02:03:07.161 INFO:teuthology.orchestra.run.smithi059.stdout:mon.a: {} 2024-04-19T02:03:07.161 INFO:teuthology.orchestra.run.smithi059.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-19T02:03:07.174 INFO:teuthology.orchestra.run.smithi059.stdout:mon.b: {} 2024-04-19T02:03:07.174 INFO:teuthology.orchestra.run.smithi059.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-19T02:03:07.188 INFO:teuthology.orchestra.run.smithi059.stdout:mon.c: {} 2024-04-19T02:03:07.189 INFO:teuthology.orchestra.run.smithi059.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-19T02:03:07.205 DEBUG:teuthology.orchestra.run.smithi081:> type ceph-objectstore-tool 2024-04-19T02:03:07.211 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-19T02:03:07.211 DEBUG:teuthology.orchestra.run.smithi059:> type ceph-objectstore-tool 2024-04-19T02:03:07.215 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-19T02:03:07.216 DEBUG:teuthology.orchestra.run.smithi179:> type ceph-objectstore-tool 2024-04-19T02:03:07.221 INFO:teuthology.orchestra.run.smithi179.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-19T02:03:07.221 INFO:teuthology.run_tasks:Running task rados... 2024-04-19T02:03:07.233 INFO:tasks.rados:Beginning rados... 2024-04-19T02:03:07.233 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-04-19T02:03:07.241 INFO:tasks.rados:joining rados 2024-04-19T02:03:07.241 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-04-19T02:03:07.241 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-19T02:03:07.241 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-19T02:03:07.241 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-19T02:03:07.241 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.046226 2024-04-19T02:03:07.241 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 0.04622616155645953 2024-04-19T02:03:07.242 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-04-19T02:03:07.242 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-04-19T02:03:07.242 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T02:03:07.243 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-04-19T02:03:07.243 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-19T02:03:07.243 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-04-19T02:03:07.243 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-19T02:03:07.243 INFO:tasks.rados:clients are ['client.0'] 2024-04-19T02:03:07.244 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set lrcprofile crush-failure-domain=osd k=4 l=3 m=2 name=lrcprofile plugin=lrc 2024-04-19T02:03:07.258 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:07.259+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:07.343 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:07.343+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:07.383 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:07.383 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:03:07.384 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-04-19T02:03:07.384 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:07.406 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-19T02:03:07.444 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:07.447+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:07.466 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-04-19T02:03:07.466 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T02:03:07.498 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-04-19T02:03:07.498 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T02:03:07.530 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-04-19T02:03:07.530 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T02:03:07.545 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:07.543+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:07.559 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:07.559 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:07.563 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-04-19T02:03:07.563 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T02:03:07.574 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-19T02:03:07.598 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-04-19T02:03:07.598 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T02:03:07.632 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-04-19T02:03:07.632 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T02:03:07.645 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:07.647+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:07.666 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-04-19T02:03:07.667 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T02:03:07.700 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-04-19T02:03:07.701 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T02:03:07.706 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:07.706 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:07.719 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-19T02:03:07.721 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-04-19T02:03:07.722 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T02:03:07.741 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-04-19T02:03:07.742 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T02:03:07.746 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:07.747+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:07.762 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-04-19T02:03:07.763 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T02:03:07.783 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-04-19T02:03:07.784 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T02:03:07.832 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:07.832 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:07.845 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-19T02:03:07.847 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:07.847+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:07.948 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:07.947+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:07.994 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:07.995 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:08.008 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-19T02:03:08.050 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:08.051+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:08.151 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:03:08.151+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:03:08.171 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:08.171 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:08.184 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-19T02:03:08.251 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:08.251+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:08.303 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:08.304 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:08.316 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-19T02:03:08.358 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:08.359+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:08.453 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:08.455+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:08.476 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:08.477 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:08.490 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-19T02:03:08.555 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:08.555+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:08.578 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is set 2024-04-19T02:03:08.578 INFO:teuthology.orchestra.run.smithi059.stderr:set osd.2 primary-affinity to 0.0462262 (830292) 2024-04-19T02:03:08.613 INFO:tasks.rados:starting run 0 out of 1 2024-04-19T02:03:08.613 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-04-19T02:03:08.613 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 16 erasure lrcprofile 2024-04-19T02:03:08.623 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:08.623 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:08.637 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-19T02:03:08.655 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:03:08.655+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:03:08.756 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:08.759+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:08.792 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:08.792 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:08.805 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-19T02:03:08.857 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:08.859+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:08.915 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:08.915 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:08.928 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-19T02:03:08.959 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:08.959+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:09.044 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:09.044 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:09.058 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-19T02:03:09.060 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:09.059+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:09.160 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:09.159+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:09.188 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:09.188 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:09.202 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-19T02:03:09.261 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:09.263+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:09.324 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:09.324 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:09.338 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-19T02:03:09.362 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:09.363+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:09.462 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:09.463+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:09.501 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:09.501 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:09.513 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-19T02:03:09.563 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:09.567+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:09.664 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:09.663+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:09.685 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:09.685 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:09.699 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-19T02:03:09.765 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:09.767+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:09.801 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T02:03:09.815 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:09.816 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:09.829 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-19T02:03:09.866 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:09.867+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:09.952 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:09.952 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:09.961 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-04-19T02:03:09.961 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T02:03:09.966 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-19T02:03:09.967 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:03:09.967+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:03:09.981 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-04-19T02:03:09.981 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T02:03:10.001 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-04-19T02:03:10.002 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T02:03:10.022 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-04-19T02:03:10.022 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T02:03:10.041 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-04-19T02:03:10.042 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T02:03:10.060 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-04-19T02:03:10.060 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T02:03:10.066 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:10.067+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:10.079 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-04-19T02:03:10.079 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T02:03:10.085 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:10.085 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:10.085 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 0 2024-04-19T02:03:10.085 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:10.097 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-04-19T02:03:10.097 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T02:03:10.099 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-19T02:03:10.121 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-04-19T02:03:10.121 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T02:03:10.142 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-04-19T02:03:10.143 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T02:03:10.165 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-04-19T02:03:10.166 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T02:03:10.169 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:03:10.167+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:03:10.189 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-04-19T02:03:10.189 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T02:03:10.215 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:10.215 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:10.215 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-04-19T02:03:10.216 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-04-19T02:03:10.216 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:10.229 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-19T02:03:10.274 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:10.275+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:10.371 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:03:10.371+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:03:10.398 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:10.398 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-04-19T02:03:10.398 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-04-19T02:03:10.398 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [] 2024-04-19T02:03:10.398 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:10.412 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-19T02:03:10.472 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:10.475+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:10.573 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:10.575+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:10.576 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:10.576 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:10.577 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 0 2024-04-19T02:03:10.577 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:10.588 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'unique_pool_0' created 2024-04-19T02:03:10.591 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-19T02:03:10.602 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-19T02:03:10.621 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-04-19T02:03:10.674 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:10.675+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:10.716 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:10.717 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:10.717 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-04-19T02:03:10.717 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-04-19T02:03:10.717 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:10.730 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-19T02:03:10.775 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:10.775+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:10.851 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:10.851 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-04-19T02:03:10.851 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-04-19T02:03:10.851 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [] 2024-04-19T02:03:10.851 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:10.866 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-19T02:03:10.876 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:10.875+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:10.977 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:10.979+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:10.997 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:10.998 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:03:10.998 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-04-19T02:03:10.998 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:11.013 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-19T02:03:11.078 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:11.079+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:11.179 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:11.179+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:11.186 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:11.186 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:03:11.186 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-04-19T02:03:11.186 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-04-19T02:03:11.186 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:11.200 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-19T02:03:11.284 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:11.287+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:11.370 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:11.370 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-04-19T02:03:11.370 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-04-19T02:03:11.370 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-04-19T02:03:11.370 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.370 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:11.370 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:02:59.539411+0000", 2024-04-19T02:03:11.371 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 11.83076784, 2024-04-19T02:03:11.371 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.43600685099999997, 2024-04-19T02:03:11.371 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:11.371 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:11.371 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:11.371 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:11.371 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:11.371 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:11.371 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 1 2024-04-19T02:03:11.371 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.372 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:11.372 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.372 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:11.372 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539411+0000", 2024-04-19T02:03:11.372 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.372 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.372 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.372 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:11.372 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539411+0000", 2024-04-19T02:03:11.372 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.372 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.373 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.373 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:11.373 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539413+0000", 2024-04-19T02:03:11.373 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3970000000000001e-06 2024-04-19T02:03:11.373 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.373 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.373 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:11.373 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539429+0000", 2024-04-19T02:03:11.373 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5153e-05 2024-04-19T02:03:11.373 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.373 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.373 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:11.373 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539450+0000", 2024-04-19T02:03:11.374 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1243e-05 2024-04-19T02:03:11.374 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.374 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.374 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:11.374 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539470+0000", 2024-04-19T02:03:11.374 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0017000000000002e-05 2024-04-19T02:03:11.374 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.374 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.374 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:11.374 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539605+0000", 2024-04-19T02:03:11.374 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013489200000000001 2024-04-19T02:03:11.374 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.375 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.375 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-04-19T02:03:11.375 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539633+0000", 2024-04-19T02:03:11.375 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8113e-05 2024-04-19T02:03:11.375 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.375 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.375 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:11.375 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.972415+0000", 2024-04-19T02:03:11.375 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.43278235300000001 2024-04-19T02:03:11.375 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.375 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.375 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:11.375 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.972942+0000", 2024-04-19T02:03:11.375 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000526379 2024-04-19T02:03:11.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.376 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.376 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:11.376 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.973384+0000", 2024-04-19T02:03:11.376 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00044265 2024-04-19T02:03:11.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.376 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.376 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:11.376 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.974795+0000", 2024-04-19T02:03:11.376 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001410592 2024-04-19T02:03:11.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.376 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.376 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:11.377 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.975249+0000", 2024-04-19T02:03:11.377 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00045376999999999999 2024-04-19T02:03:11.377 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.377 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.377 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:11.377 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.975346+0000", 2024-04-19T02:03:11.377 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7916000000000002e-05 2024-04-19T02:03:11.377 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.377 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.377 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:11.377 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.975418+0000", 2024-04-19T02:03:11.377 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1376000000000002e-05 2024-04-19T02:03:11.377 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.378 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:11.378 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.378 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.378 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.378 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:11.378 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.034302+0000", 2024-04-19T02:03:11.378 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 11.335876793000001, 2024-04-19T02:03:11.378 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012683028000000001, 2024-04-19T02:03:11.378 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:11.378 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:11.378 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:11.379 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:11.379 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:11.379 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:11.379 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 35 2024-04-19T02:03:11.379 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.379 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:11.379 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.379 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:11.379 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034302+0000", 2024-04-19T02:03:11.379 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.379 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.379 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.380 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:11.380 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034302+0000", 2024-04-19T02:03:11.380 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.380 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.380 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.380 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:11.380 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034303+0000", 2024-04-19T02:03:11.380 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4e-07 2024-04-19T02:03:11.380 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.380 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.380 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:11.380 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034443+0000", 2024-04-19T02:03:11.380 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014014499999999999 2024-04-19T02:03:11.380 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.380 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.381 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:11.381 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034459+0000", 2024-04-19T02:03:11.381 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6096999999999999e-05 2024-04-19T02:03:11.381 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.381 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.381 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:11.381 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034470+0000", 2024-04-19T02:03:11.381 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0533000000000001e-05 2024-04-19T02:03:11.381 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.381 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.381 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:11.381 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.042549+0000", 2024-04-19T02:03:11.381 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0080795309999999992 2024-04-19T02:03:11.381 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.381 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.382 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:11.382 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.042789+0000", 2024-04-19T02:03:11.382 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00023983399999999999 2024-04-19T02:03:11.382 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.382 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.382 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:11.382 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.043193+0000", 2024-04-19T02:03:11.382 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00040417699999999998 2024-04-19T02:03:11.382 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.382 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.382 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:11.382 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.046637+0000", 2024-04-19T02:03:11.382 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0034438709999999998 2024-04-19T02:03:11.382 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.383 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.383 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:11.383 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.046726+0000", 2024-04-19T02:03:11.383 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8622000000000006e-05 2024-04-19T02:03:11.383 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.383 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.383 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:11.383 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.046864+0000", 2024-04-19T02:03:11.383 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000138553 2024-04-19T02:03:11.383 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.383 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.383 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:11.383 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.046985+0000", 2024-04-19T02:03:11.383 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000120825 2024-04-19T02:03:11.383 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.384 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:11.384 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.384 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.384 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.384 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:11.384 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.034507+0000", 2024-04-19T02:03:11.384 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 11.335671389, 2024-04-19T02:03:11.384 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012727749, 2024-04-19T02:03:11.384 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:11.384 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:11.384 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:11.384 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:11.384 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:11.384 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:11.385 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 36 2024-04-19T02:03:11.385 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.385 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:11.385 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.385 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:11.385 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034507+0000", 2024-04-19T02:03:11.385 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.385 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.385 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.385 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:11.385 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034507+0000", 2024-04-19T02:03:11.385 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.385 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.385 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.386 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:11.386 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034508+0000", 2024-04-19T02:03:11.386 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0640000000000001e-06 2024-04-19T02:03:11.386 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.386 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.386 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:11.386 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034516+0000", 2024-04-19T02:03:11.386 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8199999999999997e-06 2024-04-19T02:03:11.386 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.386 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.386 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:11.386 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034528+0000", 2024-04-19T02:03:11.386 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1996e-05 2024-04-19T02:03:11.386 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.387 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.387 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:11.387 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034539+0000", 2024-04-19T02:03:11.387 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.098e-05 2024-04-19T02:03:11.387 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.387 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.387 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:11.387 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.044335+0000", 2024-04-19T02:03:11.387 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0097961309999999996 2024-04-19T02:03:11.387 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.387 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.387 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:11.387 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.044476+0000", 2024-04-19T02:03:11.388 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014055900000000001 2024-04-19T02:03:11.388 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.388 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.388 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:11.388 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.044678+0000", 2024-04-19T02:03:11.388 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000202229 2024-04-19T02:03:11.388 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.388 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.388 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:11.388 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.047076+0000", 2024-04-19T02:03:11.388 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0023975540000000001 2024-04-19T02:03:11.388 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.388 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.389 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:11.389 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.047096+0000", 2024-04-19T02:03:11.389 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0373999999999998e-05 2024-04-19T02:03:11.389 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.389 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.389 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:11.389 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.047168+0000", 2024-04-19T02:03:11.389 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1915999999999994e-05 2024-04-19T02:03:11.389 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.390 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.390 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:11.390 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.047235+0000", 2024-04-19T02:03:11.390 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7125999999999994e-05 2024-04-19T02:03:11.390 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.390 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:11.390 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.390 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.390 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.390 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:11.390 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.034553+0000", 2024-04-19T02:03:11.390 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 11.335625438999999, 2024-04-19T02:03:11.390 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013521706, 2024-04-19T02:03:11.391 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:11.391 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:11.391 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:11.391 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:11.391 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:11.391 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:11.391 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 37 2024-04-19T02:03:11.391 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.391 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:11.391 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.391 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:11.391 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034553+0000", 2024-04-19T02:03:11.391 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.392 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.392 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.392 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:11.392 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034553+0000", 2024-04-19T02:03:11.392 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.392 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.392 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.392 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:11.392 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034554+0000", 2024-04-19T02:03:11.392 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0699999999999996e-07 2024-04-19T02:03:11.392 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.392 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.393 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:11.393 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034562+0000", 2024-04-19T02:03:11.393 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7570000000000004e-06 2024-04-19T02:03:11.393 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.393 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.393 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:11.393 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034572+0000", 2024-04-19T02:03:11.393 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6730000000000004e-06 2024-04-19T02:03:11.393 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.393 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.393 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:11.394 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034580+0000", 2024-04-19T02:03:11.394 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5900000000000008e-06 2024-04-19T02:03:11.394 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.394 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.394 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:11.394 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.045799+0000", 2024-04-19T02:03:11.394 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.01121861 2024-04-19T02:03:11.394 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.394 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.394 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:11.394 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.045949+0000", 2024-04-19T02:03:11.394 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000150245 2024-04-19T02:03:11.394 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.394 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.395 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:11.395 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.046152+0000", 2024-04-19T02:03:11.395 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020245899999999999 2024-04-19T02:03:11.395 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.395 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.395 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:11.395 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.047253+0000", 2024-04-19T02:03:11.395 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001101545 2024-04-19T02:03:11.395 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.395 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.395 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:11.395 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.047800+0000", 2024-04-19T02:03:11.395 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00054726199999999997 2024-04-19T02:03:11.396 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.396 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.396 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:11.396 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.047952+0000", 2024-04-19T02:03:11.396 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015121199999999999 2024-04-19T02:03:11.396 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.396 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.396 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:11.396 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.048075+0000", 2024-04-19T02:03:11.396 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012354600000000001 2024-04-19T02:03:11.396 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.396 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:11.396 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.396 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.396 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.397 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:11.397 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.134096+0000", 2024-04-19T02:03:11.397 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 11.236082524, 2024-04-19T02:03:11.397 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015770572, 2024-04-19T02:03:11.397 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:11.397 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:11.397 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:11.397 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:11.397 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:11.397 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:11.397 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 87 2024-04-19T02:03:11.397 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.397 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:11.397 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.397 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:11.398 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.134096+0000", 2024-04-19T02:03:11.398 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.398 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.398 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.398 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:11.398 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.134096+0000", 2024-04-19T02:03:11.398 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.398 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.398 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.398 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:11.398 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.134097+0000", 2024-04-19T02:03:11.398 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2699999999999999e-07 2024-04-19T02:03:11.398 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.399 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.399 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:11.399 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.134228+0000", 2024-04-19T02:03:11.399 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013088499999999999 2024-04-19T02:03:11.399 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.399 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.399 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:11.399 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.134240+0000", 2024-04-19T02:03:11.399 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1667e-05 2024-04-19T02:03:11.399 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.399 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.399 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:11.399 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.134248+0000", 2024-04-19T02:03:11.399 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4800000000000001e-06 2024-04-19T02:03:11.400 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.400 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.400 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:11.400 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.139866+0000", 2024-04-19T02:03:11.400 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0056177400000000004 2024-04-19T02:03:11.400 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.400 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.400 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:11.400 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.140004+0000", 2024-04-19T02:03:11.401 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013788599999999999 2024-04-19T02:03:11.401 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.401 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.401 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:11.401 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.140207+0000", 2024-04-19T02:03:11.401 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020337099999999999 2024-04-19T02:03:11.401 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.401 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.402 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:11.402 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.142576+0000", 2024-04-19T02:03:11.402 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0023685989999999999 2024-04-19T02:03:11.402 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.402 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.402 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:11.402 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.149584+0000", 2024-04-19T02:03:11.402 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0070081859999999996 2024-04-19T02:03:11.402 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.402 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.402 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:11.402 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.149754+0000", 2024-04-19T02:03:11.402 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017020100000000001 2024-04-19T02:03:11.402 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.403 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.403 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:11.403 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.149867+0000", 2024-04-19T02:03:11.403 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011283 2024-04-19T02:03:11.403 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.403 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:11.403 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.403 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.403 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.403 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:11.403 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.134262+0000", 2024-04-19T02:03:11.403 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 11.235916448999999, 2024-04-19T02:03:11.403 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015881170999999999, 2024-04-19T02:03:11.403 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:11.404 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:11.404 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:11.404 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:11.404 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:11.404 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:11.404 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 88 2024-04-19T02:03:11.404 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.404 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:11.404 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.404 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:11.404 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.134262+0000", 2024-04-19T02:03:11.404 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.404 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.404 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.405 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:11.405 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.134262+0000", 2024-04-19T02:03:11.405 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.405 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.405 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.405 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:11.405 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.134263+0000", 2024-04-19T02:03:11.405 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8999999999999995e-07 2024-04-19T02:03:11.405 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.405 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.405 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:11.405 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.134270+0000", 2024-04-19T02:03:11.405 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.187e-06 2024-04-19T02:03:11.405 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.405 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.406 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:11.406 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.134280+0000", 2024-04-19T02:03:11.406 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2070000000000001e-06 2024-04-19T02:03:11.406 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.406 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.406 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:11.406 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.134289+0000", 2024-04-19T02:03:11.406 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2960000000000004e-06 2024-04-19T02:03:11.406 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.406 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.406 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:11.406 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.141715+0000", 2024-04-19T02:03:11.406 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0074262069999999998 2024-04-19T02:03:11.407 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.407 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.407 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:11.407 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.141839+0000", 2024-04-19T02:03:11.407 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012360499999999999 2024-04-19T02:03:11.407 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.407 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.407 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:11.407 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.142041+0000", 2024-04-19T02:03:11.407 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020266899999999999 2024-04-19T02:03:11.407 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.407 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.407 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:11.408 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.143179+0000", 2024-04-19T02:03:11.408 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0011378880000000001 2024-04-19T02:03:11.408 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.408 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.408 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:11.408 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.149987+0000", 2024-04-19T02:03:11.408 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0068079270000000001 2024-04-19T02:03:11.408 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.408 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.408 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:11.408 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.150078+0000", 2024-04-19T02:03:11.408 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0748999999999995e-05 2024-04-19T02:03:11.408 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.408 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.408 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:11.409 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.150144+0000", 2024-04-19T02:03:11.409 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5545999999999999e-05 2024-04-19T02:03:11.409 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.409 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:11.409 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.409 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.409 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.409 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:11.409 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.313453+0000", 2024-04-19T02:03:11.409 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 11.056725582, 2024-04-19T02:03:11.409 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014007538999999999, 2024-04-19T02:03:11.409 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:11.409 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:11.409 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:11.409 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:11.410 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:11.410 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:11.410 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 112 2024-04-19T02:03:11.410 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.410 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:11.410 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.410 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:11.410 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313453+0000", 2024-04-19T02:03:11.410 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.410 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.410 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.410 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:11.410 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313453+0000", 2024-04-19T02:03:11.410 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.411 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.411 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.411 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:11.411 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313455+0000", 2024-04-19T02:03:11.411 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3570000000000001e-06 2024-04-19T02:03:11.411 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.411 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.411 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:11.411 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313720+0000", 2024-04-19T02:03:11.411 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00026500500000000001 2024-04-19T02:03:11.411 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.411 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.411 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:11.411 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313741+0000", 2024-04-19T02:03:11.412 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1365999999999998e-05 2024-04-19T02:03:11.412 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.412 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.412 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:11.412 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313761+0000", 2024-04-19T02:03:11.412 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0100000000000001e-05 2024-04-19T02:03:11.412 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.412 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.412 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:11.412 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.322587+0000", 2024-04-19T02:03:11.412 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0088254990000000005 2024-04-19T02:03:11.412 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.412 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.412 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:11.412 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.322724+0000", 2024-04-19T02:03:11.413 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013711600000000001 2024-04-19T02:03:11.413 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.413 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.413 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:11.413 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.322929+0000", 2024-04-19T02:03:11.413 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020514799999999999 2024-04-19T02:03:11.413 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.413 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.413 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:11.413 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.327208+0000", 2024-04-19T02:03:11.413 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0042794469999999996 2024-04-19T02:03:11.413 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.413 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.413 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:11.413 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.327318+0000", 2024-04-19T02:03:11.414 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011006900000000001 2024-04-19T02:03:11.414 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.414 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.414 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:11.414 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.327401+0000", 2024-04-19T02:03:11.414 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2206000000000005e-05 2024-04-19T02:03:11.414 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.414 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.414 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:11.414 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.327461+0000", 2024-04-19T02:03:11.415 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0226000000000002e-05 2024-04-19T02:03:11.415 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.415 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:11.415 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.415 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.415 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.415 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:11.415 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.313793+0000", 2024-04-19T02:03:11.416 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 11.056385887999999, 2024-04-19T02:03:11.416 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016785001000000001, 2024-04-19T02:03:11.416 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:11.416 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:11.416 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:11.416 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:11.416 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:11.416 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:11.416 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 113 2024-04-19T02:03:11.416 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.416 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:11.416 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.417 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:11.417 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313793+0000", 2024-04-19T02:03:11.417 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.417 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.417 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.417 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:11.417 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313793+0000", 2024-04-19T02:03:11.417 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.417 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.417 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.417 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:11.417 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313795+0000", 2024-04-19T02:03:11.418 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7740000000000001e-06 2024-04-19T02:03:11.418 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.418 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.418 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:11.418 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313806+0000", 2024-04-19T02:03:11.418 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1508999999999999e-05 2024-04-19T02:03:11.418 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.418 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.418 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:11.418 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313825+0000", 2024-04-19T02:03:11.419 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8807000000000001e-05 2024-04-19T02:03:11.419 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.419 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.419 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:11.419 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313840+0000", 2024-04-19T02:03:11.419 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.483e-05 2024-04-19T02:03:11.420 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.420 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.420 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:11.420 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.323657+0000", 2024-04-19T02:03:11.420 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0098173779999999999 2024-04-19T02:03:11.420 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.420 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.421 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:11.421 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.323775+0000", 2024-04-19T02:03:11.421 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000117706 2024-04-19T02:03:11.421 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.421 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.421 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:11.422 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.323962+0000", 2024-04-19T02:03:11.422 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00018726499999999999 2024-04-19T02:03:11.422 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.422 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.422 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:11.422 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.328438+0000", 2024-04-19T02:03:11.423 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0044756379999999997 2024-04-19T02:03:11.423 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.423 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.423 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:11.423 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.330417+0000", 2024-04-19T02:03:11.423 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001978998 2024-04-19T02:03:11.424 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.424 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.424 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:11.424 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.330516+0000", 2024-04-19T02:03:11.424 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8752999999999995e-05 2024-04-19T02:03:11.424 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.425 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.425 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:11.425 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.330578+0000", 2024-04-19T02:03:11.425 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2342999999999996e-05 2024-04-19T02:03:11.425 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.425 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:11.426 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.426 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.426 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.426 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:11.426 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.313863+0000", 2024-04-19T02:03:11.426 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 11.056316038, 2024-04-19T02:03:11.426 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017815952999999999, 2024-04-19T02:03:11.427 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:11.427 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:11.427 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:11.427 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:11.427 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:11.428 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:11.428 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 114 2024-04-19T02:03:11.428 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.428 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:11.428 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.428 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:11.428 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313863+0000", 2024-04-19T02:03:11.428 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.428 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.429 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.429 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:11.429 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313863+0000", 2024-04-19T02:03:11.429 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.429 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.429 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.429 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:11.429 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313864+0000", 2024-04-19T02:03:11.429 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.403e-06 2024-04-19T02:03:11.429 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.429 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.430 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:11.430 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313875+0000", 2024-04-19T02:03:11.430 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0444e-05 2024-04-19T02:03:11.430 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.430 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.430 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:11.431 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313893+0000", 2024-04-19T02:03:11.431 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8278999999999999e-05 2024-04-19T02:03:11.431 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.431 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.431 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:11.431 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313906+0000", 2024-04-19T02:03:11.432 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3553999999999999e-05 2024-04-19T02:03:11.432 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.432 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.432 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:11.432 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.324841+0000", 2024-04-19T02:03:11.432 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010934442000000001 2024-04-19T02:03:11.432 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.433 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.433 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:11.433 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.325027+0000", 2024-04-19T02:03:11.433 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000185995 2024-04-19T02:03:11.433 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.433 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.434 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:11.434 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.325253+0000", 2024-04-19T02:03:11.434 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000226185 2024-04-19T02:03:11.434 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.434 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.435 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:11.435 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.328457+0000", 2024-04-19T02:03:11.435 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0032043750000000002 2024-04-19T02:03:11.435 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.435 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.435 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:11.436 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.331509+0000", 2024-04-19T02:03:11.436 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003051607 2024-04-19T02:03:11.436 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.436 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.436 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:11.436 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.331610+0000", 2024-04-19T02:03:11.437 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000101366 2024-04-19T02:03:11.437 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.437 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.437 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:11.437 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.331679+0000", 2024-04-19T02:03:11.437 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8303e-05 2024-04-19T02:03:11.437 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.438 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:11.438 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.438 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.438 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.438 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:11.438 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.313928+0000", 2024-04-19T02:03:11.439 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 11.056250765, 2024-04-19T02:03:11.439 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017995457999999999, 2024-04-19T02:03:11.439 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:11.439 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:11.439 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:11.440 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:11.440 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:11.440 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:11.440 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 115 2024-04-19T02:03:11.440 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.440 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:11.441 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.441 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:11.441 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313928+0000", 2024-04-19T02:03:11.441 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.441 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.441 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.442 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:11.442 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313928+0000", 2024-04-19T02:03:11.442 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.442 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.442 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.442 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:11.442 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313929+0000", 2024-04-19T02:03:11.442 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.319e-06 2024-04-19T02:03:11.443 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.443 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.443 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:11.443 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314134+0000", 2024-04-19T02:03:11.443 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020471899999999999 2024-04-19T02:03:11.443 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.443 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.443 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:11.444 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314157+0000", 2024-04-19T02:03:11.444 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.27e-05 2024-04-19T02:03:11.444 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.444 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.444 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:11.444 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314174+0000", 2024-04-19T02:03:11.444 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.694e-05 2024-04-19T02:03:11.444 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.445 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.445 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:11.445 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.325978+0000", 2024-04-19T02:03:11.445 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011804385000000001 2024-04-19T02:03:11.445 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.445 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.445 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:11.445 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.326092+0000", 2024-04-19T02:03:11.446 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000113916 2024-04-19T02:03:11.446 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.446 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.446 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:11.446 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.326279+0000", 2024-04-19T02:03:11.446 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000186632 2024-04-19T02:03:11.446 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.447 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.447 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:11.447 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.328477+0000", 2024-04-19T02:03:11.447 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0021981660000000001 2024-04-19T02:03:11.447 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.447 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.447 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:11.447 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.331786+0000", 2024-04-19T02:03:11.447 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003309055 2024-04-19T02:03:11.447 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.447 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.447 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:11.447 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.331864+0000", 2024-04-19T02:03:11.447 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8382999999999998e-05 2024-04-19T02:03:11.447 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.448 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.448 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:11.448 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.331923+0000", 2024-04-19T02:03:11.448 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9243000000000002e-05 2024-04-19T02:03:11.448 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.448 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:11.448 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.448 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.448 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.448 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:11.448 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.314202+0000", 2024-04-19T02:03:11.448 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 11.055976748000001, 2024-04-19T02:03:11.448 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017941301999999999, 2024-04-19T02:03:11.449 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:11.449 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:11.449 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:11.449 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:11.449 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:11.449 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:11.449 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 116 2024-04-19T02:03:11.449 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.449 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:11.449 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.449 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:11.450 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314202+0000", 2024-04-19T02:03:11.450 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.450 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.450 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.450 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:11.450 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314202+0000", 2024-04-19T02:03:11.450 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.450 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.450 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.450 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:11.450 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314204+0000", 2024-04-19T02:03:11.450 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7540000000000001e-06 2024-04-19T02:03:11.450 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.451 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.451 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:11.451 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314215+0000", 2024-04-19T02:03:11.451 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1345999999999999e-05 2024-04-19T02:03:11.451 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.451 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.451 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:11.451 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314233+0000", 2024-04-19T02:03:11.451 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7487e-05 2024-04-19T02:03:11.451 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.451 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.451 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:11.451 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314249+0000", 2024-04-19T02:03:11.451 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6107e-05 2024-04-19T02:03:11.452 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.452 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.452 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:11.452 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.327551+0000", 2024-04-19T02:03:11.452 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013301840000000001 2024-04-19T02:03:11.452 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.452 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.452 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:11.452 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.327676+0000", 2024-04-19T02:03:11.452 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000125716 2024-04-19T02:03:11.452 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.452 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.452 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:11.452 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.327874+0000", 2024-04-19T02:03:11.452 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00019817500000000001 2024-04-19T02:03:11.453 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.453 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.453 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:11.453 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.330619+0000", 2024-04-19T02:03:11.453 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0027450159999999999 2024-04-19T02:03:11.453 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.453 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.453 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:11.453 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.332017+0000", 2024-04-19T02:03:11.453 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0013973659999999999 2024-04-19T02:03:11.453 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.453 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.453 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:11.453 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.332090+0000", 2024-04-19T02:03:11.453 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2889e-05 2024-04-19T02:03:11.454 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.454 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.454 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:11.454 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.332143+0000", 2024-04-19T02:03:11.454 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3606000000000001e-05 2024-04-19T02:03:11.454 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.454 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:11.454 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.454 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.454 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.454 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:11.454 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.314271+0000", 2024-04-19T02:03:11.454 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 11.055907938000001, 2024-04-19T02:03:11.454 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020466839000000001, 2024-04-19T02:03:11.455 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:11.455 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:11.455 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:11.455 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:11.455 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:11.455 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:11.455 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 117 2024-04-19T02:03:11.455 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.455 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:11.455 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.455 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:11.455 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314271+0000", 2024-04-19T02:03:11.455 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.455 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.455 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.456 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:11.456 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314271+0000", 2024-04-19T02:03:11.456 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.456 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.456 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.456 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:11.456 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314272+0000", 2024-04-19T02:03:11.456 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.514e-06 2024-04-19T02:03:11.456 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.456 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.456 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:11.456 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314284+0000", 2024-04-19T02:03:11.456 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1836e-05 2024-04-19T02:03:11.456 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.456 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.457 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:11.457 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314303+0000", 2024-04-19T02:03:11.457 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8536e-05 2024-04-19T02:03:11.457 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.457 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.457 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:11.457 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314317+0000", 2024-04-19T02:03:11.457 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4209999999999999e-05 2024-04-19T02:03:11.457 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.457 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.457 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:11.457 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.328529+0000", 2024-04-19T02:03:11.457 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014211926999999999 2024-04-19T02:03:11.457 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.458 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.458 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:11.458 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.328658+0000", 2024-04-19T02:03:11.458 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000128763 2024-04-19T02:03:11.458 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.458 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.458 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:11.458 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.328864+0000", 2024-04-19T02:03:11.458 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020656099999999999 2024-04-19T02:03:11.458 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.458 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.458 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:11.458 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.331309+0000", 2024-04-19T02:03:11.458 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0024444879999999999 2024-04-19T02:03:11.459 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.459 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.459 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:11.459 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.334581+0000", 2024-04-19T02:03:11.459 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0032721450000000002 2024-04-19T02:03:11.459 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.459 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.459 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:11.459 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.334672+0000", 2024-04-19T02:03:11.459 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1267000000000006e-05 2024-04-19T02:03:11.459 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.459 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.459 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:11.459 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.334738+0000", 2024-04-19T02:03:11.459 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5592000000000005e-05 2024-04-19T02:03:11.460 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.460 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:11.460 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.460 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.460 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.460 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:11.460 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.791344+0000", 2024-04-19T02:03:11.460 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 10.578834950999999, 2024-04-19T02:03:11.460 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015697229, 2024-04-19T02:03:11.460 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:11.460 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:11.460 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:11.460 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:11.460 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:11.460 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:11.461 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 135 2024-04-19T02:03:11.461 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.461 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:11.461 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.461 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:11.461 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791344+0000", 2024-04-19T02:03:11.461 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.461 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.461 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.461 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:11.461 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791344+0000", 2024-04-19T02:03:11.461 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.461 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.461 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.461 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:11.462 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791345+0000", 2024-04-19T02:03:11.462 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7000000000000003e-07 2024-04-19T02:03:11.462 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.462 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.462 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:11.462 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791351+0000", 2024-04-19T02:03:11.462 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8100000000000003e-06 2024-04-19T02:03:11.462 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.462 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.462 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:11.462 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791360+0000", 2024-04-19T02:03:11.462 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3170000000000008e-06 2024-04-19T02:03:11.462 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.463 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.463 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:11.463 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791368+0000", 2024-04-19T02:03:11.463 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5800000000000003e-06 2024-04-19T02:03:11.463 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.463 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.463 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:11.463 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.800383+0000", 2024-04-19T02:03:11.463 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0090151569999999993 2024-04-19T02:03:11.463 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.463 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.463 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:11.463 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.800622+0000", 2024-04-19T02:03:11.463 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000239564 2024-04-19T02:03:11.464 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.464 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.464 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:11.464 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.800969+0000", 2024-04-19T02:03:11.464 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00034647500000000001 2024-04-19T02:03:11.464 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.464 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.464 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:11.464 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.803767+0000", 2024-04-19T02:03:11.464 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002798488 2024-04-19T02:03:11.464 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.464 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.464 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:11.464 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.806778+0000", 2024-04-19T02:03:11.464 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0030110430000000001 2024-04-19T02:03:11.465 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.465 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.465 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:11.465 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.806913+0000", 2024-04-19T02:03:11.465 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000134332 2024-04-19T02:03:11.465 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.465 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.465 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:11.465 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.807041+0000", 2024-04-19T02:03:11.465 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000128493 2024-04-19T02:03:11.465 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.465 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:11.465 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.465 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.465 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.466 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:11.466 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.791379+0000", 2024-04-19T02:03:11.466 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 10.578800298000001, 2024-04-19T02:03:11.466 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.01942953, 2024-04-19T02:03:11.466 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:11.466 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:11.466 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:11.466 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:11.466 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:11.466 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:11.466 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 136 2024-04-19T02:03:11.466 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.466 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:11.467 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.467 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:11.467 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791379+0000", 2024-04-19T02:03:11.467 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.467 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.467 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.467 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:11.467 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791379+0000", 2024-04-19T02:03:11.467 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.467 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.467 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.467 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:11.467 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791379+0000", 2024-04-19T02:03:11.468 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4000000000000001e-07 2024-04-19T02:03:11.468 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.468 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.468 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:11.468 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791387+0000", 2024-04-19T02:03:11.468 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9160000000000001e-06 2024-04-19T02:03:11.468 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.468 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.468 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:11.468 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791396+0000", 2024-04-19T02:03:11.468 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2939999999999994e-06 2024-04-19T02:03:11.468 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.468 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.468 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:11.469 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791404+0000", 2024-04-19T02:03:11.469 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6329999999999996e-06 2024-04-19T02:03:11.469 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.469 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.469 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:11.469 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.802771+0000", 2024-04-19T02:03:11.469 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011366619 2024-04-19T02:03:11.469 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.469 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.469 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:11.469 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.802918+0000", 2024-04-19T02:03:11.469 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014683900000000001 2024-04-19T02:03:11.469 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.469 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.470 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:11.470 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.803129+0000", 2024-04-19T02:03:11.470 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021161899999999999 2024-04-19T02:03:11.470 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.470 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.470 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:11.470 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.806732+0000", 2024-04-19T02:03:11.470 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0036026679999999998 2024-04-19T02:03:11.470 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.470 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.470 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:11.470 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.810544+0000", 2024-04-19T02:03:11.470 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0038118470000000002 2024-04-19T02:03:11.470 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.471 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.471 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:11.471 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.810676+0000", 2024-04-19T02:03:11.471 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013222900000000001 2024-04-19T02:03:11.471 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.471 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.471 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:11.471 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.810808+0000", 2024-04-19T02:03:11.471 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013212599999999999 2024-04-19T02:03:11.471 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.471 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:11.471 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.471 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.472 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.472 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:11.472 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.791415+0000", 2024-04-19T02:03:11.472 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 10.578763628000001, 2024-04-19T02:03:11.472 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019711117, 2024-04-19T02:03:11.472 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:11.472 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:11.472 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:11.472 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:11.472 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:11.472 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:11.472 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 137 2024-04-19T02:03:11.472 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.473 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:11.473 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.473 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:11.473 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791415+0000", 2024-04-19T02:03:11.473 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.473 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.473 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.473 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:11.473 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791415+0000", 2024-04-19T02:03:11.473 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.473 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.473 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.474 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:11.474 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791416+0000", 2024-04-19T02:03:11.474 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.23e-07 2024-04-19T02:03:11.474 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.474 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.474 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:11.474 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791584+0000", 2024-04-19T02:03:11.474 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016797599999999999 2024-04-19T02:03:11.474 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.474 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.474 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:11.474 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791600+0000", 2024-04-19T02:03:11.474 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5945999999999999e-05 2024-04-19T02:03:11.474 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.475 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.475 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:11.475 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791610+0000", 2024-04-19T02:03:11.475 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.01e-05 2024-04-19T02:03:11.475 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.475 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.475 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:11.475 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.803873+0000", 2024-04-19T02:03:11.475 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012262848999999999 2024-04-19T02:03:11.475 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.475 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.475 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:11.475 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.803995+0000", 2024-04-19T02:03:11.476 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012215899999999999 2024-04-19T02:03:11.476 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.476 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.476 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:11.476 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.804217+0000", 2024-04-19T02:03:11.476 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000222201 2024-04-19T02:03:11.476 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.476 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.476 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:11.476 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.808399+0000", 2024-04-19T02:03:11.476 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0041816910000000004 2024-04-19T02:03:11.476 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.476 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.476 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:11.477 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.810864+0000", 2024-04-19T02:03:11.477 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0024651650000000001 2024-04-19T02:03:11.477 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.477 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.477 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:11.477 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.811011+0000", 2024-04-19T02:03:11.477 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014738199999999999 2024-04-19T02:03:11.477 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.477 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.477 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:11.477 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.811126+0000", 2024-04-19T02:03:11.477 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000114925 2024-04-19T02:03:11.477 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.477 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:11.478 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.478 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.478 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.478 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:11.478 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.791625+0000", 2024-04-19T02:03:11.478 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 10.578553492999999, 2024-04-19T02:03:11.478 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019753863999999999, 2024-04-19T02:03:11.478 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:11.478 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:11.478 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:11.478 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:11.478 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:11.478 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:11.479 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 138 2024-04-19T02:03:11.479 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.479 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:11.479 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.479 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:11.479 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791625+0000", 2024-04-19T02:03:11.479 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.479 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.479 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.479 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:11.479 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791625+0000", 2024-04-19T02:03:11.479 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.479 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.480 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.480 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:11.480 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791626+0000", 2024-04-19T02:03:11.480 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8599999999999996e-07 2024-04-19T02:03:11.480 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.480 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.480 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:11.480 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791632+0000", 2024-04-19T02:03:11.480 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7200000000000003e-06 2024-04-19T02:03:11.480 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.480 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.480 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:11.480 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791641+0000", 2024-04-19T02:03:11.480 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4299999999999995e-06 2024-04-19T02:03:11.480 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.480 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.481 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:11.481 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791649+0000", 2024-04-19T02:03:11.481 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8129999999999996e-06 2024-04-19T02:03:11.481 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.481 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.481 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:11.481 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.805144+0000", 2024-04-19T02:03:11.481 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013494920000000001 2024-04-19T02:03:11.481 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.481 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.481 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:11.481 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.805387+0000", 2024-04-19T02:03:11.481 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000243188 2024-04-19T02:03:11.481 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.481 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.482 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:11.482 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.805776+0000", 2024-04-19T02:03:11.482 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00038853999999999999 2024-04-19T02:03:11.482 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.482 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.482 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:11.482 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.808536+0000", 2024-04-19T02:03:11.482 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0027598760000000001 2024-04-19T02:03:11.482 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.482 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.482 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:11.482 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.811151+0000", 2024-04-19T02:03:11.482 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0026147560000000002 2024-04-19T02:03:11.482 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.483 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.483 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:11.483 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.811275+0000", 2024-04-19T02:03:11.483 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012435199999999999 2024-04-19T02:03:11.483 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.483 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.483 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:11.483 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.811379+0000", 2024-04-19T02:03:11.483 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000104283 2024-04-19T02:03:11.483 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.483 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:11.483 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.483 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.483 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.484 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:11.484 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.791660+0000", 2024-04-19T02:03:11.484 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 10.578518467, 2024-04-19T02:03:11.484 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021026688000000002, 2024-04-19T02:03:11.484 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:11.484 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:11.484 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:11.484 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:11.484 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:11.484 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:11.484 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 139 2024-04-19T02:03:11.484 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.484 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:11.484 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.485 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:11.485 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791660+0000", 2024-04-19T02:03:11.485 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.485 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.485 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.485 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:11.485 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791660+0000", 2024-04-19T02:03:11.485 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.485 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.485 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.485 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:11.485 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791661+0000", 2024-04-19T02:03:11.485 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2699999999999999e-07 2024-04-19T02:03:11.485 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.486 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.486 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:11.486 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791666+0000", 2024-04-19T02:03:11.486 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4160000000000003e-06 2024-04-19T02:03:11.486 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.486 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.486 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:11.486 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791676+0000", 2024-04-19T02:03:11.486 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1169999999999993e-06 2024-04-19T02:03:11.486 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.486 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.486 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:11.486 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791683+0000", 2024-04-19T02:03:11.486 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8369999999999997e-06 2024-04-19T02:03:11.487 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.487 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.487 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:11.487 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.807160+0000", 2024-04-19T02:03:11.487 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.01547656 2024-04-19T02:03:11.487 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.487 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.487 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:11.487 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.807408+0000", 2024-04-19T02:03:11.487 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000248485 2024-04-19T02:03:11.487 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.487 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.487 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:11.487 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.807622+0000", 2024-04-19T02:03:11.488 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021381199999999999 2024-04-19T02:03:11.488 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.488 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.488 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:11.488 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812397+0000", 2024-04-19T02:03:11.488 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0047750229999999998 2024-04-19T02:03:11.488 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.488 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.488 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:11.488 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812434+0000", 2024-04-19T02:03:11.488 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6869000000000002e-05 2024-04-19T02:03:11.488 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.488 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.488 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:11.488 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812571+0000", 2024-04-19T02:03:11.489 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013649199999999999 2024-04-19T02:03:11.489 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.489 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.489 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:11.489 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812687+0000", 2024-04-19T02:03:11.489 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011635 2024-04-19T02:03:11.489 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.489 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:11.489 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.489 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.489 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.489 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:11.489 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.791694+0000", 2024-04-19T02:03:11.489 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 10.578484547, 2024-04-19T02:03:11.490 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021303507999999999, 2024-04-19T02:03:11.490 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:11.490 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:11.490 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:11.490 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:11.490 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:11.490 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:11.490 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 140 2024-04-19T02:03:11.490 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.490 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:11.490 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.491 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:11.491 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791694+0000", 2024-04-19T02:03:11.491 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.491 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.491 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.491 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:11.491 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791694+0000", 2024-04-19T02:03:11.491 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.491 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.491 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.491 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:11.491 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791695+0000", 2024-04-19T02:03:11.491 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2699999999999999e-07 2024-04-19T02:03:11.491 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.491 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.492 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:11.492 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791829+0000", 2024-04-19T02:03:11.492 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013436500000000001 2024-04-19T02:03:11.492 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.492 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.492 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:11.492 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791841+0000", 2024-04-19T02:03:11.492 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1707e-05 2024-04-19T02:03:11.492 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.492 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.492 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:11.492 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791850+0000", 2024-04-19T02:03:11.492 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5799999999999992e-06 2024-04-19T02:03:11.492 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.492 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.493 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:11.493 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.808635+0000", 2024-04-19T02:03:11.493 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016785133000000001 2024-04-19T02:03:11.493 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.493 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.493 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:11.493 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.808862+0000", 2024-04-19T02:03:11.493 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022693799999999999 2024-04-19T02:03:11.493 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.493 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.493 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:11.493 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.809242+0000", 2024-04-19T02:03:11.494 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00038053400000000003 2024-04-19T02:03:11.494 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.494 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.494 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:11.494 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812709+0000", 2024-04-19T02:03:11.494 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0034666929999999999 2024-04-19T02:03:11.494 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.494 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.494 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:11.494 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812796+0000", 2024-04-19T02:03:11.494 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6877000000000002e-05 2024-04-19T02:03:11.494 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.494 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.494 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:11.495 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812910+0000", 2024-04-19T02:03:11.495 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000113929 2024-04-19T02:03:11.495 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.495 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.495 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:11.495 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812998+0000", 2024-04-19T02:03:11.495 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8024999999999998e-05 2024-04-19T02:03:11.495 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.495 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:11.495 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.495 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.495 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.495 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:11.495 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.791863+0000", 2024-04-19T02:03:11.495 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 10.578315738000001, 2024-04-19T02:03:11.496 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021620092, 2024-04-19T02:03:11.496 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:11.496 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:11.496 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:11.496 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:11.496 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:11.496 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:11.496 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 141 2024-04-19T02:03:11.496 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.496 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:11.496 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.496 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:11.496 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791863+0000", 2024-04-19T02:03:11.496 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.496 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.497 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.497 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:11.497 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791863+0000", 2024-04-19T02:03:11.497 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.497 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.497 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.497 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:11.497 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791864+0000", 2024-04-19T02:03:11.497 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0599999999999999e-07 2024-04-19T02:03:11.497 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.497 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.497 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:11.497 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791871+0000", 2024-04-19T02:03:11.497 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1040000000000001e-06 2024-04-19T02:03:11.497 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.498 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.498 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:11.498 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791881+0000", 2024-04-19T02:03:11.498 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4329999999999993e-06 2024-04-19T02:03:11.498 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.498 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.498 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:11.498 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791888+0000", 2024-04-19T02:03:11.498 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7360000000000001e-06 2024-04-19T02:03:11.498 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.498 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.498 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:11.498 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.811476+0000", 2024-04-19T02:03:11.498 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019588186 2024-04-19T02:03:11.499 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.499 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.499 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:11.499 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.811595+0000", 2024-04-19T02:03:11.499 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011901200000000001 2024-04-19T02:03:11.499 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.499 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.499 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:11.499 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.811793+0000", 2024-04-19T02:03:11.499 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000197958 2024-04-19T02:03:11.499 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.499 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.499 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:11.499 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.813066+0000", 2024-04-19T02:03:11.500 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001272884 2024-04-19T02:03:11.500 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.500 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.500 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:11.500 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.813356+0000", 2024-04-19T02:03:11.500 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028986399999999997 2024-04-19T02:03:11.500 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.500 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.500 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:11.500 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.813430+0000", 2024-04-19T02:03:11.500 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3716999999999999e-05 2024-04-19T02:03:11.500 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.500 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.500 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:11.501 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.813483+0000", 2024-04-19T02:03:11.501 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3291999999999997e-05 2024-04-19T02:03:11.501 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.501 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:11.501 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.501 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.501 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.501 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:11.501 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:01.293321+0000", 2024-04-19T02:03:11.501 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 10.076858266, 2024-04-19T02:03:11.501 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011980067, 2024-04-19T02:03:11.501 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:11.502 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:11.502 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:11.502 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:11.502 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:11.502 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:11.502 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 165 2024-04-19T02:03:11.502 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.502 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:11.502 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.502 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:11.502 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:01.293321+0000", 2024-04-19T02:03:11.502 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.503 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.503 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.503 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:11.503 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:01.293321+0000", 2024-04-19T02:03:11.503 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:11.503 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.503 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.503 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:11.503 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:01.293321+0000", 2024-04-19T02:03:11.503 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4199999999999996e-07 2024-04-19T02:03:11.503 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.503 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.504 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:11.504 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:01.293327+0000", 2024-04-19T02:03:11.504 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7100000000000004e-06 2024-04-19T02:03:11.504 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.504 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.504 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:11.504 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:01.293332+0000", 2024-04-19T02:03:11.504 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3979999999999998e-06 2024-04-19T02:03:11.504 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.504 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.504 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:11.504 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:01.293336+0000", 2024-04-19T02:03:11.504 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9530000000000001e-06 2024-04-19T02:03:11.504 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.504 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.504 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:11.505 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:01.301596+0000", 2024-04-19T02:03:11.505 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0082593649999999994 2024-04-19T02:03:11.505 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.505 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.505 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:11.505 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:01.301713+0000", 2024-04-19T02:03:11.505 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000117846 2024-04-19T02:03:11.505 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.505 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.505 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:11.505 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:01.301886+0000", 2024-04-19T02:03:11.505 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017283899999999999 2024-04-19T02:03:11.505 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.505 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.505 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:11.506 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:01.302616+0000", 2024-04-19T02:03:11.506 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00072964899999999997 2024-04-19T02:03:11.506 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.506 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.506 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:11.506 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:01.305249+0000", 2024-04-19T02:03:11.506 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0026333279999999999 2024-04-19T02:03:11.506 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.506 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.506 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:11.506 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:01.305275+0000", 2024-04-19T02:03:11.506 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6055999999999999e-05 2024-04-19T02:03:11.506 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:11.506 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:11.507 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:11.507 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:01.305301+0000", 2024-04-19T02:03:11.507 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5381e-05 2024-04-19T02:03:11.507 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.507 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:11.507 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.507 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:11.507 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:11.507 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:11.509 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-19T02:03:11.509 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:11.507+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:11.597 INFO:teuthology.orchestra.run.smithi059.stderr:nodeep-scrub is set 2024-04-19T02:03:11.597 INFO:teuthology.orchestra.run.smithi059.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-04-19T02:03:11.630 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:11.631+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:11.696 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:11.697 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:03:11.697 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-04-19T02:03:11.697 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:11.711 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-19T02:03:11.712 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:11.711+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:11.811 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:11.811+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:11.830 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:11.830 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:03:11.830 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-04-19T02:03:11.830 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-04-19T02:03:11.830 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:11.844 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-19T02:03:11.912 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:11.911+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:11.964 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:11.964 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-04-19T02:03:11.965 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-04-19T02:03:11.965 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [] 2024-04-19T02:03:11.965 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:11.979 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-19T02:03:12.013 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:12.011+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:12.114 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:12.115+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:12.146 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:12.146 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:03:12.146 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-04-19T02:03:12.146 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:12.160 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-19T02:03:12.210 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T02:03:12.214 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:12.215+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:12.284 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:12.284 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:03:12.284 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-04-19T02:03:12.284 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-04-19T02:03:12.284 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:12.298 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-19T02:03:12.315 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:12.315+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:12.369 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-04-19T02:03:12.369 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T02:03:12.386 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-04-19T02:03:12.386 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T02:03:12.403 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-04-19T02:03:12.403 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T02:03:12.416 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:12.415+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:12.421 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-04-19T02:03:12.422 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T02:03:12.425 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:12.425 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-04-19T02:03:12.425 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-04-19T02:03:12.425 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [] 2024-04-19T02:03:12.425 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:12.439 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-19T02:03:12.442 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-04-19T02:03:12.442 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T02:03:12.463 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-04-19T02:03:12.464 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T02:03:12.485 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-04-19T02:03:12.485 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T02:03:12.506 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-04-19T02:03:12.506 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T02:03:12.526 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-04-19T02:03:12.526 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T02:03:12.532 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:12.535+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:12.546 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-04-19T02:03:12.546 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T02:03:12.567 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-04-19T02:03:12.567 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T02:03:12.589 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-04-19T02:03:12.590 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T02:03:12.611 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:12.612 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:12.626 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-19T02:03:12.636 DEBUG:teuthology.orchestra.run.smithi059:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0 2024-04-19T02:03:12.654 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:12.655+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:12.673 INFO:tasks.rados.rados.0.smithi059.stdout:adding op weight read -> 100 2024-04-19T02:03:12.674 INFO:tasks.rados.rados.0.smithi059.stdout:adding op weight delete -> 50 2024-04-19T02:03:12.674 INFO:tasks.rados.rados.0.smithi059.stdout:adding op weight snap_create -> 50 2024-04-19T02:03:12.674 INFO:tasks.rados.rados.0.smithi059.stdout:adding op weight snap_remove -> 50 2024-04-19T02:03:12.674 INFO:tasks.rados.rados.0.smithi059.stdout:adding op weight rollback -> 50 2024-04-19T02:03:12.674 INFO:tasks.rados.rados.0.smithi059.stdout:adding op weight setattr -> 25 2024-04-19T02:03:12.674 INFO:tasks.rados.rados.0.smithi059.stdout:adding op weight rmattr -> 25 2024-04-19T02:03:12.674 INFO:tasks.rados.rados.0.smithi059.stdout:adding op weight copy_from -> 50 2024-04-19T02:03:12.674 INFO:tasks.rados.rados.0.smithi059.stdout:adding op weight append -> 100 2024-04-19T02:03:12.674 INFO:tasks.rados.rados.0.smithi059.stdout:ceph version 19.0.0-2369-g1a6a3e74 (1a6a3e742fd81c6d70be70552499bff767652c32) squid (dev) 2024-04-19T02:03:12.674 INFO:tasks.rados.rados.0.smithi059.stdout:Configuration: 2024-04-19T02:03:12.674 INFO:tasks.rados.rados.0.smithi059.stdout: Number of operations: 400 2024-04-19T02:03:12.674 INFO:tasks.rados.rados.0.smithi059.stdout: Number of objects: 50 2024-04-19T02:03:12.675 INFO:tasks.rados.rados.0.smithi059.stdout: Max in flight operations: 16 2024-04-19T02:03:12.675 INFO:tasks.rados.rados.0.smithi059.stdout: Object size (in bytes): 4000000 2024-04-19T02:03:12.675 INFO:tasks.rados.rados.0.smithi059.stdout: Write stride min: 400000 2024-04-19T02:03:12.675 INFO:tasks.rados.rados.0.smithi059.stdout: Write stride max: 800000 2024-04-19T02:03:12.718 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:12.719+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:12.793 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:12.793 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:12.808 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-19T02:03:12.853 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:12.851+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:12.954 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:12.955+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:12.974 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:12.975 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:12.989 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-19T02:03:13.054 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:13.055+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:13.123 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:13.123 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:13.137 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-19T02:03:13.155 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:13.155+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:13.256 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:13.259+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:13.258 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:13.259 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:13.275 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-19T02:03:13.358 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:13.359+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:13.401 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:13.401 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:13.416 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-19T02:03:13.458 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:03:13.459+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:03:13.537 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:13.537 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:13.551 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-19T02:03:13.560 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:13.559+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:13.603 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-19T02:03:13.603 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-19T02:03:13.604 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-04-19T02:03:13.604 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T02:03:13.611 INFO:tasks.rados.rados.0.smithi059.stdout:1: write initial oid 1 2024-04-19T02:03:13.611 INFO:tasks.rados.rados.0.smithi059.stdout:1: seq_num 0 ranges {0=32768} 2024-04-19T02:03:13.611 INFO:tasks.rados.rados.0.smithi059.stdout:1: writing smithi05930959-1 from 0 to 32768 tid 1 2024-04-19T02:03:13.611 INFO:tasks.rados.rados.0.smithi059.stdout:2: write initial oid 2 2024-04-19T02:03:13.611 INFO:tasks.rados.rados.0.smithi059.stdout:2: seq_num 1 ranges {0=32768} 2024-04-19T02:03:13.611 INFO:tasks.rados.rados.0.smithi059.stdout:2: writing smithi05930959-2 from 0 to 32768 tid 1 2024-04-19T02:03:13.611 INFO:tasks.rados.rados.0.smithi059.stdout:3: write initial oid 3 2024-04-19T02:03:13.612 INFO:tasks.rados.rados.0.smithi059.stdout:3: seq_num 2 ranges {0=49152} 2024-04-19T02:03:13.612 INFO:tasks.rados.rados.0.smithi059.stdout:3: writing smithi05930959-3 from 0 to 49152 tid 1 2024-04-19T02:03:13.612 INFO:tasks.rados.rados.0.smithi059.stdout:4: write initial oid 4 2024-04-19T02:03:13.612 INFO:tasks.rados.rados.0.smithi059.stdout:4: seq_num 3 ranges {0=65536} 2024-04-19T02:03:13.612 INFO:tasks.rados.rados.0.smithi059.stdout:4: writing smithi05930959-4 from 0 to 65536 tid 1 2024-04-19T02:03:13.612 INFO:tasks.rados.rados.0.smithi059.stdout:5: write initial oid 5 2024-04-19T02:03:13.612 INFO:tasks.rados.rados.0.smithi059.stdout:5: seq_num 4 ranges {0=81920} 2024-04-19T02:03:13.613 INFO:tasks.rados.rados.0.smithi059.stdout:5: writing smithi05930959-5 from 0 to 81920 tid 1 2024-04-19T02:03:13.613 INFO:tasks.rados.rados.0.smithi059.stdout:6: write initial oid 6 2024-04-19T02:03:13.613 INFO:tasks.rados.rados.0.smithi059.stdout:6: seq_num 5 ranges {0=98304} 2024-04-19T02:03:13.617 INFO:tasks.rados.rados.0.smithi059.stdout:6: writing smithi05930959-6 from 0 to 98304 tid 1 2024-04-19T02:03:13.617 INFO:tasks.rados.rados.0.smithi059.stdout:7: write initial oid 7 2024-04-19T02:03:13.617 INFO:tasks.rados.rados.0.smithi059.stdout:7: seq_num 6 ranges {0=114688} 2024-04-19T02:03:13.617 INFO:tasks.rados.rados.0.smithi059.stdout:7: writing smithi05930959-7 from 0 to 114688 tid 1 2024-04-19T02:03:13.617 INFO:tasks.rados.rados.0.smithi059.stdout:8: write initial oid 8 2024-04-19T02:03:13.617 INFO:tasks.rados.rados.0.smithi059.stdout:8: seq_num 7 ranges {0=131072} 2024-04-19T02:03:13.617 INFO:tasks.rados.rados.0.smithi059.stdout:8: writing smithi05930959-8 from 0 to 131072 tid 1 2024-04-19T02:03:13.617 INFO:tasks.rados.rados.0.smithi059.stdout:9: write initial oid 9 2024-04-19T02:03:13.617 INFO:tasks.rados.rados.0.smithi059.stdout:9: seq_num 8 ranges {0=147456} 2024-04-19T02:03:13.618 INFO:tasks.rados.rados.0.smithi059.stdout:9: writing smithi05930959-9 from 0 to 147456 tid 1 2024-04-19T02:03:13.618 INFO:tasks.rados.rados.0.smithi059.stdout:10: write initial oid 10 2024-04-19T02:03:13.618 INFO:tasks.rados.rados.0.smithi059.stdout:10: seq_num 9 ranges {0=163840} 2024-04-19T02:03:13.620 INFO:tasks.rados.rados.0.smithi059.stdout:10: writing smithi05930959-10 from 0 to 163840 tid 1 2024-04-19T02:03:13.620 INFO:tasks.rados.rados.0.smithi059.stdout:11: write initial oid 11 2024-04-19T02:03:13.620 INFO:tasks.rados.rados.0.smithi059.stdout:11: seq_num 10 ranges {0=180224} 2024-04-19T02:03:13.621 INFO:tasks.rados.rados.0.smithi059.stdout:11: writing smithi05930959-11 from 0 to 180224 tid 1 2024-04-19T02:03:13.622 INFO:tasks.rados.rados.0.smithi059.stdout:12: write initial oid 12 2024-04-19T02:03:13.622 INFO:tasks.rados.rados.0.smithi059.stdout:12: seq_num 11 ranges {0=196608} 2024-04-19T02:03:13.623 INFO:tasks.rados.rados.0.smithi059.stdout:12: writing smithi05930959-12 from 0 to 196608 tid 1 2024-04-19T02:03:13.623 INFO:tasks.rados.rados.0.smithi059.stdout:13: write initial oid 13 2024-04-19T02:03:13.623 INFO:tasks.rados.rados.0.smithi059.stdout:13: seq_num 12 ranges {0=212992} 2024-04-19T02:03:13.625 INFO:tasks.rados.rados.0.smithi059.stdout:13: writing smithi05930959-13 from 0 to 212992 tid 1 2024-04-19T02:03:13.625 INFO:tasks.rados.rados.0.smithi059.stdout:14: write initial oid 14 2024-04-19T02:03:13.625 INFO:tasks.rados.rados.0.smithi059.stdout:14: seq_num 13 ranges {0=229376} 2024-04-19T02:03:13.626 INFO:tasks.rados.rados.0.smithi059.stdout:14: writing smithi05930959-14 from 0 to 229376 tid 1 2024-04-19T02:03:13.627 INFO:tasks.rados.rados.0.smithi059.stdout:15: write initial oid 15 2024-04-19T02:03:13.627 INFO:tasks.rados.rados.0.smithi059.stdout:15: seq_num 14 ranges {0=245760} 2024-04-19T02:03:13.629 INFO:tasks.rados.rados.0.smithi059.stdout:15: writing smithi05930959-15 from 0 to 245760 tid 1 2024-04-19T02:03:13.629 INFO:tasks.rados.rados.0.smithi059.stdout:16: write initial oid 16 2024-04-19T02:03:13.629 INFO:tasks.rados.rados.0.smithi059.stdout:16: seq_num 15 ranges {0=262144} 2024-04-19T02:03:13.632 INFO:tasks.rados.rados.0.smithi059.stdout:16: writing smithi05930959-16 from 0 to 262144 tid 1 2024-04-19T02:03:13.632 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-19T02:03:13.637 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-19T02:03:13.639 INFO:tasks.rados.rados.0.smithi059.stdout:5: finishing write tid 1 to smithi05930959-5 2024-04-19T02:03:13.639 INFO:tasks.rados.rados.0.smithi059.stdout:5: finishing write tid 2 to smithi05930959-5 2024-04-19T02:03:13.644 INFO:tasks.rados.rados.0.smithi059.stdout:2: finishing write tid 1 to smithi05930959-2 2024-04-19T02:03:13.648 INFO:tasks.rados.rados.0.smithi059.stdout:6: finishing write tid 1 to smithi05930959-6 2024-04-19T02:03:13.648 INFO:tasks.rados.rados.0.smithi059.stdout:10: finishing write tid 1 to smithi05930959-10 2024-04-19T02:03:13.648 INFO:tasks.rados.rados.0.smithi059.stdout:3: finishing write tid 1 to smithi05930959-3 2024-04-19T02:03:13.649 INFO:tasks.rados.rados.0.smithi059.stdout:11: finishing write tid 1 to smithi05930959-11 2024-04-19T02:03:13.649 INFO:tasks.rados.rados.0.smithi059.stdout:11: finishing write tid 2 to smithi05930959-11 2024-04-19T02:03:13.650 INFO:tasks.rados.rados.0.smithi059.stdout:10: finishing write tid 2 to smithi05930959-10 2024-04-19T02:03:13.655 INFO:tasks.rados.rados.0.smithi059.stdout:11: finishing write tid 3 to smithi05930959-11 2024-04-19T02:03:13.655 INFO:tasks.rados.rados.0.smithi059.stdout:11: oid 11 updating version 0 to 2 2024-04-19T02:03:13.655 INFO:tasks.rados.rados.0.smithi059.stdout:11: oid 11 version 2 is already newer than 1 2024-04-19T02:03:13.655 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 11 v 2 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-19T02:03:13.655 INFO:tasks.rados.rados.0.smithi059.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-04-19T02:03:13.655 INFO:tasks.rados.rados.0.smithi059.stdout:11: done (15 left) 2024-04-19T02:03:13.655 INFO:tasks.rados.rados.0.smithi059.stdout:17: write initial oid 17 2024-04-19T02:03:13.655 INFO:tasks.rados.rados.0.smithi059.stdout:17: seq_num 16 ranges {0=278528} 2024-04-19T02:03:13.656 INFO:tasks.rados.rados.0.smithi059.stdout:17: writing smithi05930959-17 from 0 to 278528 tid 1 2024-04-19T02:03:13.656 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-19T02:03:13.656 INFO:tasks.rados.rados.0.smithi059.stdout:4: finishing write tid 1 to smithi05930959-4 2024-04-19T02:03:13.656 INFO:tasks.rados.rados.0.smithi059.stdout:4: finishing write tid 2 to smithi05930959-4 2024-04-19T02:03:13.656 INFO:tasks.rados.rados.0.smithi059.stdout:6: finishing write tid 2 to smithi05930959-6 2024-04-19T02:03:13.656 INFO:tasks.rados.rados.0.smithi059.stdout:8: finishing write tid 1 to smithi05930959-8 2024-04-19T02:03:13.657 INFO:tasks.rados.rados.0.smithi059.stdout:2: finishing write tid 2 to smithi05930959-2 2024-04-19T02:03:13.658 INFO:tasks.rados.rados.0.smithi059.stdout:15: finishing write tid 1 to smithi05930959-15 2024-04-19T02:03:13.660 INFO:tasks.rados.rados.0.smithi059.stdout:10: finishing write tid 3 to smithi05930959-10 2024-04-19T02:03:13.660 INFO:tasks.rados.rados.0.smithi059.stdout:10: oid 10 updating version 0 to 1 2024-04-19T02:03:13.660 INFO:tasks.rados.rados.0.smithi059.stdout:10: oid 10 updating version 1 to 2 2024-04-19T02:03:13.660 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 10 v 2 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-19T02:03:13.660 INFO:tasks.rados.rados.0.smithi059.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-04-19T02:03:13.661 INFO:tasks.rados.rados.0.smithi059.stdout:3: finishing write tid 2 to smithi05930959-3 2024-04-19T02:03:13.661 INFO:tasks.rados.rados.0.smithi059.stdout:8: finishing write tid 2 to smithi05930959-8 2024-04-19T02:03:13.661 INFO:tasks.rados.rados.0.smithi059.stdout:10: done (15 left) 2024-04-19T02:03:13.661 INFO:tasks.rados.rados.0.smithi059.stdout:18: write initial oid 18 2024-04-19T02:03:13.661 INFO:tasks.rados.rados.0.smithi059.stdout:18: seq_num 17 ranges {0=294912} 2024-04-19T02:03:13.661 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:13.659+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:13.662 INFO:tasks.rados.rados.0.smithi059.stdout:18: writing smithi05930959-18 from 0 to 294912 tid 1 2024-04-19T02:03:13.662 INFO:tasks.rados.rados.0.smithi059.stdout:9: finishing write tid 1 to smithi05930959-9 2024-04-19T02:03:13.662 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-19T02:03:13.663 INFO:tasks.rados.rados.0.smithi059.stdout:9: finishing write tid 2 to smithi05930959-9 2024-04-19T02:03:13.663 INFO:tasks.rados.rados.0.smithi059.stdout:16: finishing write tid 1 to smithi05930959-16 2024-04-19T02:03:13.663 INFO:tasks.rados.rados.0.smithi059.stdout:16: finishing write tid 2 to smithi05930959-16 2024-04-19T02:03:13.665 INFO:tasks.rados.rados.0.smithi059.stdout:3: finishing write tid 3 to smithi05930959-3 2024-04-19T02:03:13.665 INFO:tasks.rados.rados.0.smithi059.stdout:3: oid 3 updating version 0 to 2 2024-04-19T02:03:13.665 INFO:tasks.rados.rados.0.smithi059.stdout:3: oid 3 version 2 is already newer than 1 2024-04-19T02:03:13.666 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 3 v 2 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-19T02:03:13.666 INFO:tasks.rados.rados.0.smithi059.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-04-19T02:03:13.666 INFO:tasks.rados.rados.0.smithi059.stdout:3: done (15 left) 2024-04-19T02:03:13.666 INFO:tasks.rados.rados.0.smithi059.stdout:19: write initial oid 19 2024-04-19T02:03:13.666 INFO:tasks.rados.rados.0.smithi059.stdout:19: seq_num 18 ranges {0=311296} 2024-04-19T02:03:13.668 INFO:tasks.rados.rados.0.smithi059.stdout:19: writing smithi05930959-19 from 0 to 311296 tid 1 2024-04-19T02:03:13.668 INFO:tasks.rados.rados.0.smithi059.stdout:17: finishing write tid 1 to smithi05930959-17 2024-04-19T02:03:13.668 INFO:tasks.rados.rados.0.smithi059.stdout:4: finishing write tid 3 to smithi05930959-4 2024-04-19T02:03:13.668 INFO:tasks.rados.rados.0.smithi059.stdout:4: oid 4 updating version 0 to 1 2024-04-19T02:03:13.668 INFO:tasks.rados.rados.0.smithi059.stdout:4: oid 4 updating version 1 to 2 2024-04-19T02:03:13.668 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 4 v 2 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-19T02:03:13.668 INFO:tasks.rados.rados.0.smithi059.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-04-19T02:03:13.668 INFO:tasks.rados.rados.0.smithi059.stdout:17: finishing write tid 2 to smithi05930959-17 2024-04-19T02:03:13.669 INFO:tasks.rados.rados.0.smithi059.stdout:1: finishing write tid 1 to smithi05930959-1 2024-04-19T02:03:13.669 INFO:tasks.rados.rados.0.smithi059.stdout:1: finishing write tid 2 to smithi05930959-1 2024-04-19T02:03:13.669 INFO:tasks.rados.rados.0.smithi059.stdout:14: finishing write tid 1 to smithi05930959-14 2024-04-19T02:03:13.669 INFO:tasks.rados.rados.0.smithi059.stdout:4: done (15 left) 2024-04-19T02:03:13.669 INFO:tasks.rados.rados.0.smithi059.stdout:20: write initial oid 20 2024-04-19T02:03:13.669 INFO:tasks.rados.rados.0.smithi059.stdout:20: seq_num 19 ranges {0=327680} 2024-04-19T02:03:13.671 INFO:tasks.rados.rados.0.smithi059.stdout:20: writing smithi05930959-20 from 0 to 327680 tid 1 2024-04-19T02:03:13.671 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-19T02:03:13.672 INFO:tasks.rados.rados.0.smithi059.stdout:14: finishing write tid 2 to smithi05930959-14 2024-04-19T02:03:13.672 INFO:tasks.rados.rados.0.smithi059.stdout:13: finishing write tid 1 to smithi05930959-13 2024-04-19T02:03:13.672 INFO:tasks.rados.rados.0.smithi059.stdout:13: finishing write tid 2 to smithi05930959-13 2024-04-19T02:03:13.672 INFO:tasks.rados.rados.0.smithi059.stdout:15: finishing write tid 2 to smithi05930959-15 2024-04-19T02:03:13.672 INFO:tasks.rados.rados.0.smithi059.stdout:14: finishing write tid 3 to smithi05930959-14 2024-04-19T02:03:13.672 INFO:tasks.rados.rados.0.smithi059.stdout:14: oid 14 updating version 0 to 2 2024-04-19T02:03:13.672 INFO:tasks.rados.rados.0.smithi059.stdout:14: oid 14 version 2 is already newer than 1 2024-04-19T02:03:13.672 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 14 v 2 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-19T02:03:13.672 INFO:tasks.rados.rados.0.smithi059.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-04-19T02:03:13.672 INFO:tasks.rados.rados.0.smithi059.stdout:14: done (15 left) 2024-04-19T02:03:13.672 INFO:tasks.rados.rados.0.smithi059.stdout:21: write initial oid 21 2024-04-19T02:03:13.672 INFO:tasks.rados.rados.0.smithi059.stdout:21: seq_num 20 ranges {0=344064} 2024-04-19T02:03:13.675 INFO:tasks.rados.rados.0.smithi059.stdout:21: writing smithi05930959-21 from 0 to 344064 tid 1 2024-04-19T02:03:13.675 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 16 2024-04-19T02:03:13.676 INFO:tasks.rados.rados.0.smithi059.stdout:16: finishing write tid 3 to smithi05930959-16 2024-04-19T02:03:13.676 INFO:tasks.rados.rados.0.smithi059.stdout:16: oid 16 updating version 0 to 4 2024-04-19T02:03:13.676 INFO:tasks.rados.rados.0.smithi059.stdout:16: oid 16 version 4 is already newer than 3 2024-04-19T02:03:13.676 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 16 v 4 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-19T02:03:13.676 INFO:tasks.rados.rados.0.smithi059.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-04-19T02:03:13.676 INFO:tasks.rados.rados.0.smithi059.stdout:2: finishing write tid 3 to smithi05930959-2 2024-04-19T02:03:13.676 INFO:tasks.rados.rados.0.smithi059.stdout:2: oid 2 updating version 0 to 2 2024-04-19T02:03:13.676 INFO:tasks.rados.rados.0.smithi059.stdout:2: oid 2 version 2 is already newer than 1 2024-04-19T02:03:13.676 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 2 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-19T02:03:13.676 INFO:tasks.rados.rados.0.smithi059.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-04-19T02:03:13.676 INFO:tasks.rados.rados.0.smithi059.stdout:8: finishing write tid 3 to smithi05930959-8 2024-04-19T02:03:13.676 INFO:tasks.rados.rados.0.smithi059.stdout:8: oid 8 updating version 0 to 1 2024-04-19T02:03:13.676 INFO:tasks.rados.rados.0.smithi059.stdout:8: oid 8 updating version 1 to 2 2024-04-19T02:03:13.676 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 8 v 2 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-19T02:03:13.676 INFO:tasks.rados.rados.0.smithi059.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-04-19T02:03:13.676 INFO:tasks.rados.rados.0.smithi059.stdout:5: finishing write tid 3 to smithi05930959-5 2024-04-19T02:03:13.677 INFO:tasks.rados.rados.0.smithi059.stdout:5: oid 5 updating version 0 to 1 2024-04-19T02:03:13.677 INFO:tasks.rados.rados.0.smithi059.stdout:5: oid 5 updating version 1 to 2 2024-04-19T02:03:13.677 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 5 v 2 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-19T02:03:13.677 INFO:tasks.rados.rados.0.smithi059.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-04-19T02:03:13.677 INFO:tasks.rados.rados.0.smithi059.stdout:13: finishing write tid 3 to smithi05930959-13 2024-04-19T02:03:13.677 INFO:tasks.rados.rados.0.smithi059.stdout:13: oid 13 updating version 0 to 3 2024-04-19T02:03:13.677 INFO:tasks.rados.rados.0.smithi059.stdout:13: oid 13 updating version 3 to 4 2024-04-19T02:03:13.677 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 13 v 4 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-19T02:03:13.677 INFO:tasks.rados.rados.0.smithi059.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-04-19T02:03:13.677 INFO:tasks.rados.rados.0.smithi059.stdout:9: finishing write tid 3 to smithi05930959-9 2024-04-19T02:03:13.677 INFO:tasks.rados.rados.0.smithi059.stdout:9: oid 9 updating version 0 to 3 2024-04-19T02:03:13.677 INFO:tasks.rados.rados.0.smithi059.stdout:9: oid 9 updating version 3 to 4 2024-04-19T02:03:13.677 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 9 v 4 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-19T02:03:13.677 INFO:tasks.rados.rados.0.smithi059.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-04-19T02:03:13.678 INFO:tasks.rados.rados.0.smithi059.stdout:15: finishing write tid 3 to smithi05930959-15 2024-04-19T02:03:13.678 INFO:tasks.rados.rados.0.smithi059.stdout:15: oid 15 updating version 0 to 1 2024-04-19T02:03:13.678 INFO:tasks.rados.rados.0.smithi059.stdout:15: oid 15 updating version 1 to 2 2024-04-19T02:03:13.678 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 15 v 2 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-19T02:03:13.678 INFO:tasks.rados.rados.0.smithi059.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-04-19T02:03:13.678 INFO:tasks.rados.rados.0.smithi059.stdout:12: finishing write tid 1 to smithi05930959-12 2024-04-19T02:03:13.678 INFO:tasks.rados.rados.0.smithi059.stdout:12: finishing write tid 2 to smithi05930959-12 2024-04-19T02:03:13.678 INFO:tasks.rados.rados.0.smithi059.stdout:2: done (15 left) 2024-04-19T02:03:13.678 INFO:tasks.rados.rados.0.smithi059.stdout:5: done (14 left) 2024-04-19T02:03:13.678 INFO:tasks.rados.rados.0.smithi059.stdout:8: done (13 left) 2024-04-19T02:03:13.678 INFO:tasks.rados.rados.0.smithi059.stdout:9: done (12 left) 2024-04-19T02:03:13.678 INFO:tasks.rados.rados.0.smithi059.stdout:13: done (11 left) 2024-04-19T02:03:13.678 INFO:tasks.rados.rados.0.smithi059.stdout:15: done (10 left) 2024-04-19T02:03:13.678 INFO:tasks.rados.rados.0.smithi059.stdout:16: done (9 left) 2024-04-19T02:03:13.678 INFO:tasks.rados.rados.0.smithi059.stdout:22: write initial oid 22 2024-04-19T02:03:13.678 INFO:tasks.rados.rados.0.smithi059.stdout:22: seq_num 21 ranges {0=360448} 2024-04-19T02:03:13.679 INFO:tasks.rados.rados.0.smithi059.stdout:22: writing smithi05930959-22 from 0 to 360448 tid 1 2024-04-19T02:03:13.679 INFO:tasks.rados.rados.0.smithi059.stdout:23: write initial oid 23 2024-04-19T02:03:13.679 INFO:tasks.rados.rados.0.smithi059.stdout:23: seq_num 22 ranges {0=376832} 2024-04-19T02:03:13.684 INFO:tasks.rados.rados.0.smithi059.stdout:23: writing smithi05930959-23 from 0 to 376832 tid 1 2024-04-19T02:03:13.685 INFO:tasks.rados.rados.0.smithi059.stdout:6: finishing write tid 3 to smithi05930959-6 2024-04-19T02:03:13.685 INFO:tasks.rados.rados.0.smithi059.stdout:6: oid 6 updating version 0 to 1 2024-04-19T02:03:13.685 INFO:tasks.rados.rados.0.smithi059.stdout:6: oid 6 updating version 1 to 2 2024-04-19T02:03:13.686 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 6 v 2 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-19T02:03:13.686 INFO:tasks.rados.rados.0.smithi059.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-04-19T02:03:13.686 INFO:tasks.rados.rados.0.smithi059.stdout:17: finishing write tid 3 to smithi05930959-17 2024-04-19T02:03:13.686 INFO:tasks.rados.rados.0.smithi059.stdout:17: oid 17 updating version 0 to 4 2024-04-19T02:03:13.686 INFO:tasks.rados.rados.0.smithi059.stdout:17: oid 17 version 4 is already newer than 3 2024-04-19T02:03:13.686 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 17 v 4 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-19T02:03:13.686 INFO:tasks.rados.rados.0.smithi059.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-04-19T02:03:13.686 INFO:tasks.rados.rados.0.smithi059.stdout:12: finishing write tid 3 to smithi05930959-12 2024-04-19T02:03:13.686 INFO:tasks.rados.rados.0.smithi059.stdout:12: oid 12 updating version 0 to 1 2024-04-19T02:03:13.686 INFO:tasks.rados.rados.0.smithi059.stdout:12: oid 12 updating version 1 to 2 2024-04-19T02:03:13.686 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 12 v 2 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-19T02:03:13.686 INFO:tasks.rados.rados.0.smithi059.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-04-19T02:03:13.686 INFO:tasks.rados.rados.0.smithi059.stdout:7: finishing write tid 1 to smithi05930959-7 2024-04-19T02:03:13.686 INFO:tasks.rados.rados.0.smithi059.stdout:7: finishing write tid 2 to smithi05930959-7 2024-04-19T02:03:13.686 INFO:tasks.rados.rados.0.smithi059.stdout:6: done (10 left) 2024-04-19T02:03:13.687 INFO:tasks.rados.rados.0.smithi059.stdout:12: done (9 left) 2024-04-19T02:03:13.687 INFO:tasks.rados.rados.0.smithi059.stdout:17: done (8 left) 2024-04-19T02:03:13.687 INFO:tasks.rados.rados.0.smithi059.stdout:24: write initial oid 24 2024-04-19T02:03:13.687 INFO:tasks.rados.rados.0.smithi059.stdout:18: finishing write tid 1 to smithi05930959-18 2024-04-19T02:03:13.687 INFO:tasks.rados.rados.0.smithi059.stdout:24: seq_num 23 ranges {0=393216} 2024-04-19T02:03:13.688 INFO:tasks.rados.rados.0.smithi059.stdout:24: writing smithi05930959-24 from 0 to 393216 tid 1 2024-04-19T02:03:13.689 INFO:tasks.rados.rados.0.smithi059.stdout:25: write initial oid 25 2024-04-19T02:03:13.689 INFO:tasks.rados.rados.0.smithi059.stdout:25: seq_num 24 ranges {0=409600} 2024-04-19T02:03:13.694 INFO:tasks.rados.rados.0.smithi059.stdout:25: writing smithi05930959-25 from 0 to 409600 tid 1 2024-04-19T02:03:13.695 INFO:tasks.rados.rados.0.smithi059.stdout:1: finishing write tid 3 to smithi05930959-1 2024-04-19T02:03:13.695 INFO:tasks.rados.rados.0.smithi059.stdout:1: oid 1 updating version 0 to 2 2024-04-19T02:03:13.695 INFO:tasks.rados.rados.0.smithi059.stdout:1: oid 1 version 2 is already newer than 1 2024-04-19T02:03:13.695 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 1 v 2 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-19T02:03:13.695 INFO:tasks.rados.rados.0.smithi059.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-04-19T02:03:13.695 INFO:tasks.rados.rados.0.smithi059.stdout:7: finishing write tid 3 to smithi05930959-7 2024-04-19T02:03:13.695 INFO:tasks.rados.rados.0.smithi059.stdout:7: oid 7 updating version 0 to 4 2024-04-19T02:03:13.695 INFO:tasks.rados.rados.0.smithi059.stdout:7: oid 7 version 4 is already newer than 3 2024-04-19T02:03:13.695 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 7 v 4 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-19T02:03:13.695 INFO:tasks.rados.rados.0.smithi059.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-04-19T02:03:13.695 INFO:tasks.rados.rados.0.smithi059.stdout:21: finishing write tid 1 to smithi05930959-21 2024-04-19T02:03:13.695 INFO:tasks.rados.rados.0.smithi059.stdout:18: finishing write tid 2 to smithi05930959-18 2024-04-19T02:03:13.695 INFO:tasks.rados.rados.0.smithi059.stdout:19: finishing write tid 1 to smithi05930959-19 2024-04-19T02:03:13.696 INFO:tasks.rados.rados.0.smithi059.stdout:20: finishing write tid 1 to smithi05930959-20 2024-04-19T02:03:13.696 INFO:tasks.rados.rados.0.smithi059.stdout:20: finishing write tid 2 to smithi05930959-20 2024-04-19T02:03:13.696 INFO:tasks.rados.rados.0.smithi059.stdout:19: finishing write tid 2 to smithi05930959-19 2024-04-19T02:03:13.696 INFO:tasks.rados.rados.0.smithi059.stdout:21: finishing write tid 2 to smithi05930959-21 2024-04-19T02:03:13.696 INFO:tasks.rados.rados.0.smithi059.stdout:22: finishing write tid 1 to smithi05930959-22 2024-04-19T02:03:13.696 INFO:tasks.rados.rados.0.smithi059.stdout:18: finishing write tid 3 to smithi05930959-18 2024-04-19T02:03:13.696 INFO:tasks.rados.rados.0.smithi059.stdout:18: oid 18 updating version 0 to 5 2024-04-19T02:03:13.696 INFO:tasks.rados.rados.0.smithi059.stdout:18: oid 18 updating version 5 to 6 2024-04-19T02:03:13.696 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-19T02:03:13.696 INFO:tasks.rados.rados.0.smithi059.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-04-19T02:03:13.696 INFO:tasks.rados.rados.0.smithi059.stdout:21: finishing write tid 3 to smithi05930959-21 2024-04-19T02:03:13.696 INFO:tasks.rados.rados.0.smithi059.stdout:21: oid 21 updating version 0 to 5 2024-04-19T02:03:13.696 INFO:tasks.rados.rados.0.smithi059.stdout:21: oid 21 updating version 5 to 6 2024-04-19T02:03:13.696 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 21 v 6 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-19T02:03:13.696 INFO:tasks.rados.rados.0.smithi059.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-04-19T02:03:13.697 INFO:tasks.rados.rados.0.smithi059.stdout:20: finishing write tid 3 to smithi05930959-20 2024-04-19T02:03:13.697 INFO:tasks.rados.rados.0.smithi059.stdout:20: oid 20 updating version 0 to 1 2024-04-19T02:03:13.697 INFO:tasks.rados.rados.0.smithi059.stdout:20: oid 20 updating version 1 to 2 2024-04-19T02:03:13.697 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 20 v 2 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-19T02:03:13.697 INFO:tasks.rados.rados.0.smithi059.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-04-19T02:03:13.697 INFO:tasks.rados.rados.0.smithi059.stdout:1: done (9 left) 2024-04-19T02:03:13.697 INFO:tasks.rados.rados.0.smithi059.stdout:7: done (8 left) 2024-04-19T02:03:13.697 INFO:tasks.rados.rados.0.smithi059.stdout:18: done (7 left) 2024-04-19T02:03:13.697 INFO:tasks.rados.rados.0.smithi059.stdout:20: done (6 left) 2024-04-19T02:03:13.697 INFO:tasks.rados.rados.0.smithi059.stdout:21: done (5 left) 2024-04-19T02:03:13.697 INFO:tasks.rados.rados.0.smithi059.stdout:26: write initial oid 26 2024-04-19T02:03:13.697 INFO:tasks.rados.rados.0.smithi059.stdout:26: seq_num 25 ranges {0=425984} 2024-04-19T02:03:13.698 INFO:tasks.rados.rados.0.smithi059.stdout:26: writing smithi05930959-26 from 0 to 425984 tid 1 2024-04-19T02:03:13.698 INFO:tasks.rados.rados.0.smithi059.stdout:27: write initial oid 27 2024-04-19T02:03:13.698 INFO:tasks.rados.rados.0.smithi059.stdout:27: seq_num 26 ranges {0=442368} 2024-04-19T02:03:13.704 INFO:tasks.rados.rados.0.smithi059.stdout:27: writing smithi05930959-27 from 0 to 442368 tid 1 2024-04-19T02:03:13.704 INFO:tasks.rados.rados.0.smithi059.stdout:28: write initial oid 28 2024-04-19T02:03:13.704 INFO:tasks.rados.rados.0.smithi059.stdout:28: seq_num 27 ranges {0=458752} 2024-04-19T02:03:13.709 INFO:tasks.rados.rados.0.smithi059.stdout:28: writing smithi05930959-28 from 0 to 458752 tid 1 2024-04-19T02:03:13.709 INFO:tasks.rados.rados.0.smithi059.stdout:29: write initial oid 29 2024-04-19T02:03:13.709 INFO:tasks.rados.rados.0.smithi059.stdout:29: seq_num 28 ranges {0=475136} 2024-04-19T02:03:13.713 INFO:tasks.rados.rados.0.smithi059.stdout:29: writing smithi05930959-29 from 0 to 475136 tid 1 2024-04-19T02:03:13.713 INFO:tasks.rados.rados.0.smithi059.stdout:30: write initial oid 30 2024-04-19T02:03:13.713 INFO:tasks.rados.rados.0.smithi059.stdout:30: seq_num 29 ranges {0=491520} 2024-04-19T02:03:13.718 INFO:tasks.rados.rados.0.smithi059.stdout:30: writing smithi05930959-30 from 0 to 491520 tid 1 2024-04-19T02:03:13.719 INFO:tasks.rados.rados.0.smithi059.stdout:22: finishing write tid 2 to smithi05930959-22 2024-04-19T02:03:13.719 INFO:tasks.rados.rados.0.smithi059.stdout:23: finishing write tid 1 to smithi05930959-23 2024-04-19T02:03:13.719 INFO:tasks.rados.rados.0.smithi059.stdout:23: finishing write tid 2 to smithi05930959-23 2024-04-19T02:03:13.719 INFO:tasks.rados.rados.0.smithi059.stdout:19: finishing write tid 3 to smithi05930959-19 2024-04-19T02:03:13.719 INFO:tasks.rados.rados.0.smithi059.stdout:19: oid 19 updating version 0 to 6 2024-04-19T02:03:13.719 INFO:tasks.rados.rados.0.smithi059.stdout:19: oid 19 version 6 is already newer than 5 2024-04-19T02:03:13.719 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 19 v 6 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-19T02:03:13.719 INFO:tasks.rados.rados.0.smithi059.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-04-19T02:03:13.719 INFO:tasks.rados.rados.0.smithi059.stdout:22: finishing write tid 3 to smithi05930959-22 2024-04-19T02:03:13.719 INFO:tasks.rados.rados.0.smithi059.stdout:22: oid 22 updating version 0 to 7 2024-04-19T02:03:13.720 INFO:tasks.rados.rados.0.smithi059.stdout:22: oid 22 updating version 7 to 8 2024-04-19T02:03:13.720 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 22 v 8 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-19T02:03:13.720 INFO:tasks.rados.rados.0.smithi059.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-04-19T02:03:13.720 INFO:tasks.rados.rados.0.smithi059.stdout:24: finishing write tid 1 to smithi05930959-24 2024-04-19T02:03:13.720 INFO:tasks.rados.rados.0.smithi059.stdout:24: finishing write tid 2 to smithi05930959-24 2024-04-19T02:03:13.720 INFO:tasks.rados.rados.0.smithi059.stdout:23: finishing write tid 3 to smithi05930959-23 2024-04-19T02:03:13.720 INFO:tasks.rados.rados.0.smithi059.stdout:23: oid 23 updating version 0 to 10 2024-04-19T02:03:13.720 INFO:tasks.rados.rados.0.smithi059.stdout:23: oid 23 version 10 is already newer than 9 2024-04-19T02:03:13.720 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 23 v 10 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-19T02:03:13.720 INFO:tasks.rados.rados.0.smithi059.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-04-19T02:03:13.720 INFO:tasks.rados.rados.0.smithi059.stdout:24: finishing write tid 3 to smithi05930959-24 2024-04-19T02:03:13.720 INFO:tasks.rados.rados.0.smithi059.stdout:24: oid 24 updating version 0 to 1 2024-04-19T02:03:13.720 INFO:tasks.rados.rados.0.smithi059.stdout:24: oid 24 updating version 1 to 2 2024-04-19T02:03:13.720 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 24 v 2 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-19T02:03:13.721 INFO:tasks.rados.rados.0.smithi059.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-04-19T02:03:13.721 INFO:tasks.rados.rados.0.smithi059.stdout:25: finishing write tid 1 to smithi05930959-25 2024-04-19T02:03:13.721 INFO:tasks.rados.rados.0.smithi059.stdout:25: finishing write tid 2 to smithi05930959-25 2024-04-19T02:03:13.721 INFO:tasks.rados.rados.0.smithi059.stdout:26: finishing write tid 1 to smithi05930959-26 2024-04-19T02:03:13.721 INFO:tasks.rados.rados.0.smithi059.stdout:25: finishing write tid 3 to smithi05930959-25 2024-04-19T02:03:13.721 INFO:tasks.rados.rados.0.smithi059.stdout:25: oid 25 updating version 0 to 3 2024-04-19T02:03:13.721 INFO:tasks.rados.rados.0.smithi059.stdout:25: oid 25 updating version 3 to 4 2024-04-19T02:03:13.721 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 25 v 4 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-19T02:03:13.721 INFO:tasks.rados.rados.0.smithi059.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-04-19T02:03:13.721 INFO:tasks.rados.rados.0.smithi059.stdout:26: finishing write tid 2 to smithi05930959-26 2024-04-19T02:03:13.721 INFO:tasks.rados.rados.0.smithi059.stdout:26: finishing write tid 3 to smithi05930959-26 2024-04-19T02:03:13.721 INFO:tasks.rados.rados.0.smithi059.stdout:26: oid 26 updating version 0 to 3 2024-04-19T02:03:13.721 INFO:tasks.rados.rados.0.smithi059.stdout:26: oid 26 updating version 3 to 4 2024-04-19T02:03:13.721 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 26 v 4 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-19T02:03:13.722 INFO:tasks.rados.rados.0.smithi059.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-04-19T02:03:13.722 INFO:tasks.rados.rados.0.smithi059.stdout:27: finishing write tid 1 to smithi05930959-27 2024-04-19T02:03:13.722 INFO:tasks.rados.rados.0.smithi059.stdout:27: finishing write tid 2 to smithi05930959-27 2024-04-19T02:03:13.722 INFO:tasks.rados.rados.0.smithi059.stdout:28: finishing write tid 1 to smithi05930959-28 2024-04-19T02:03:13.722 INFO:tasks.rados.rados.0.smithi059.stdout:28: finishing write tid 2 to smithi05930959-28 2024-04-19T02:03:13.722 INFO:tasks.rados.rados.0.smithi059.stdout:19: done (9 left) 2024-04-19T02:03:13.722 INFO:tasks.rados.rados.0.smithi059.stdout:22: done (8 left) 2024-04-19T02:03:13.722 INFO:tasks.rados.rados.0.smithi059.stdout:23: done (7 left) 2024-04-19T02:03:13.722 INFO:tasks.rados.rados.0.smithi059.stdout:24: done (6 left) 2024-04-19T02:03:13.722 INFO:tasks.rados.rados.0.smithi059.stdout:25: done (5 left) 2024-04-19T02:03:13.722 INFO:tasks.rados.rados.0.smithi059.stdout:26: done (4 left) 2024-04-19T02:03:13.722 INFO:tasks.rados.rados.0.smithi059.stdout:31: write initial oid 31 2024-04-19T02:03:13.722 INFO:tasks.rados.rados.0.smithi059.stdout:31: seq_num 30 ranges {0=507904} 2024-04-19T02:03:13.725 INFO:tasks.rados.rados.0.smithi059.stdout:31: writing smithi05930959-31 from 0 to 507904 tid 1 2024-04-19T02:03:13.725 INFO:tasks.rados.rados.0.smithi059.stdout:29: finishing write tid 1 to smithi05930959-29 2024-04-19T02:03:13.725 INFO:tasks.rados.rados.0.smithi059.stdout:27: finishing write tid 3 to smithi05930959-27 2024-04-19T02:03:13.725 INFO:tasks.rados.rados.0.smithi059.stdout:27: oid 27 updating version 0 to 12 2024-04-19T02:03:13.725 INFO:tasks.rados.rados.0.smithi059.stdout:27: oid 27 version 12 is already newer than 11 2024-04-19T02:03:13.725 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 27 v 12 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-19T02:03:13.725 INFO:tasks.rados.rados.0.smithi059.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-04-19T02:03:13.725 INFO:tasks.rados.rados.0.smithi059.stdout:28: finishing write tid 3 to smithi05930959-28 2024-04-19T02:03:13.726 INFO:tasks.rados.rados.0.smithi059.stdout:28: oid 28 updating version 0 to 1 2024-04-19T02:03:13.726 INFO:tasks.rados.rados.0.smithi059.stdout:28: oid 28 updating version 1 to 2 2024-04-19T02:03:13.726 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 28 v 2 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-19T02:03:13.726 INFO:tasks.rados.rados.0.smithi059.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-04-19T02:03:13.726 INFO:tasks.rados.rados.0.smithi059.stdout:29: finishing write tid 2 to smithi05930959-29 2024-04-19T02:03:13.726 INFO:tasks.rados.rados.0.smithi059.stdout:29: finishing write tid 3 to smithi05930959-29 2024-04-19T02:03:13.726 INFO:tasks.rados.rados.0.smithi059.stdout:29: oid 29 updating version 0 to 4 2024-04-19T02:03:13.726 INFO:tasks.rados.rados.0.smithi059.stdout:29: oid 29 version 4 is already newer than 3 2024-04-19T02:03:13.726 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 29 v 4 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-19T02:03:13.726 INFO:tasks.rados.rados.0.smithi059.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-04-19T02:03:13.726 INFO:tasks.rados.rados.0.smithi059.stdout:30: finishing write tid 1 to smithi05930959-30 2024-04-19T02:03:13.726 INFO:tasks.rados.rados.0.smithi059.stdout:27: done (4 left) 2024-04-19T02:03:13.726 INFO:tasks.rados.rados.0.smithi059.stdout:28: done (3 left) 2024-04-19T02:03:13.726 INFO:tasks.rados.rados.0.smithi059.stdout:29: done (2 left) 2024-04-19T02:03:13.726 INFO:tasks.rados.rados.0.smithi059.stdout:32: write initial oid 32 2024-04-19T02:03:13.727 INFO:tasks.rados.rados.0.smithi059.stdout:32: seq_num 31 ranges {0=524288} 2024-04-19T02:03:13.729 INFO:tasks.rados.rados.0.smithi059.stdout:32: writing smithi05930959-32 from 0 to 524288 tid 1 2024-04-19T02:03:13.730 INFO:tasks.rados.rados.0.smithi059.stdout:30: finishing write tid 2 to smithi05930959-30 2024-04-19T02:03:13.730 INFO:tasks.rados.rados.0.smithi059.stdout:30: finishing write tid 3 to smithi05930959-30 2024-04-19T02:03:13.730 INFO:tasks.rados.rados.0.smithi059.stdout:30: oid 30 updating version 0 to 3 2024-04-19T02:03:13.730 INFO:tasks.rados.rados.0.smithi059.stdout:30: oid 30 updating version 3 to 4 2024-04-19T02:03:13.731 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 30 v 4 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-19T02:03:13.731 INFO:tasks.rados.rados.0.smithi059.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-04-19T02:03:13.731 INFO:tasks.rados.rados.0.smithi059.stdout:31: finishing write tid 1 to smithi05930959-31 2024-04-19T02:03:13.731 INFO:tasks.rados.rados.0.smithi059.stdout:30: done (2 left) 2024-04-19T02:03:13.731 INFO:tasks.rados.rados.0.smithi059.stdout:33: write initial oid 33 2024-04-19T02:03:13.731 INFO:tasks.rados.rados.0.smithi059.stdout:33: seq_num 32 ranges {0=540672} 2024-04-19T02:03:13.735 INFO:tasks.rados.rados.0.smithi059.stdout:33: writing smithi05930959-33 from 0 to 540672 tid 1 2024-04-19T02:03:13.735 INFO:tasks.rados.rados.0.smithi059.stdout:31: finishing write tid 2 to smithi05930959-31 2024-04-19T02:03:13.735 INFO:tasks.rados.rados.0.smithi059.stdout:31: finishing write tid 3 to smithi05930959-31 2024-04-19T02:03:13.736 INFO:tasks.rados.rados.0.smithi059.stdout:31: oid 31 updating version 0 to 5 2024-04-19T02:03:13.736 INFO:tasks.rados.rados.0.smithi059.stdout:31: oid 31 updating version 5 to 6 2024-04-19T02:03:13.736 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 31 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-19T02:03:13.736 INFO:tasks.rados.rados.0.smithi059.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-04-19T02:03:13.736 INFO:tasks.rados.rados.0.smithi059.stdout:31: done (2 left) 2024-04-19T02:03:13.736 INFO:tasks.rados.rados.0.smithi059.stdout:34: write initial oid 34 2024-04-19T02:03:13.736 INFO:tasks.rados.rados.0.smithi059.stdout:34: seq_num 33 ranges {0=557056} 2024-04-19T02:03:13.737 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:13.737 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:13.740 INFO:tasks.rados.rados.0.smithi059.stdout:34: writing smithi05930959-34 from 0 to 557056 tid 1 2024-04-19T02:03:13.740 INFO:tasks.rados.rados.0.smithi059.stdout:32: finishing write tid 1 to smithi05930959-32 2024-04-19T02:03:13.741 INFO:tasks.rados.rados.0.smithi059.stdout:32: finishing write tid 2 to smithi05930959-32 2024-04-19T02:03:13.741 INFO:tasks.rados.rados.0.smithi059.stdout:33: finishing write tid 1 to smithi05930959-33 2024-04-19T02:03:13.741 INFO:tasks.rados.rados.0.smithi059.stdout:35: write initial oid 35 2024-04-19T02:03:13.741 INFO:tasks.rados.rados.0.smithi059.stdout:35: seq_num 34 ranges {0=573440} 2024-04-19T02:03:13.745 INFO:tasks.rados.rados.0.smithi059.stdout:35: writing smithi05930959-35 from 0 to 573440 tid 1 2024-04-19T02:03:13.746 INFO:tasks.rados.rados.0.smithi059.stdout:32: finishing write tid 3 to smithi05930959-32 2024-04-19T02:03:13.746 INFO:tasks.rados.rados.0.smithi059.stdout:32: oid 32 updating version 0 to 13 2024-04-19T02:03:13.746 INFO:tasks.rados.rados.0.smithi059.stdout:32: oid 32 updating version 13 to 14 2024-04-19T02:03:13.746 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 32 v 14 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-19T02:03:13.746 INFO:tasks.rados.rados.0.smithi059.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-04-19T02:03:13.746 INFO:tasks.rados.rados.0.smithi059.stdout:33: finishing write tid 2 to smithi05930959-33 2024-04-19T02:03:13.746 INFO:tasks.rados.rados.0.smithi059.stdout:32: done (3 left) 2024-04-19T02:03:13.746 INFO:tasks.rados.rados.0.smithi059.stdout:36: write initial oid 36 2024-04-19T02:03:13.747 INFO:tasks.rados.rados.0.smithi059.stdout:33: finishing write tid 3 to smithi05930959-33 2024-04-19T02:03:13.747 INFO:tasks.rados.rados.0.smithi059.stdout:33: oid 33 updating version 0 to 3 2024-04-19T02:03:13.747 INFO:tasks.rados.rados.0.smithi059.stdout:33: oid 33 updating version 3 to 4 2024-04-19T02:03:13.747 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 33 v 4 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-19T02:03:13.747 INFO:tasks.rados.rados.0.smithi059.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-04-19T02:03:13.747 INFO:tasks.rados.rados.0.smithi059.stdout:36: seq_num 35 ranges {0=589824} 2024-04-19T02:03:13.752 INFO:tasks.rados.rados.0.smithi059.stdout:36: writing smithi05930959-36 from 0 to 589824 tid 1 2024-04-19T02:03:13.753 INFO:tasks.rados.rados.0.smithi059.stdout:34: finishing write tid 1 to smithi05930959-34 2024-04-19T02:03:13.753 INFO:tasks.rados.rados.0.smithi059.stdout:34: finishing write tid 2 to smithi05930959-34 2024-04-19T02:03:13.753 INFO:tasks.rados.rados.0.smithi059.stdout:34: finishing write tid 3 to smithi05930959-34 2024-04-19T02:03:13.753 INFO:tasks.rados.rados.0.smithi059.stdout:34: oid 34 updating version 0 to 5 2024-04-19T02:03:13.753 INFO:tasks.rados.rados.0.smithi059.stdout:34: oid 34 updating version 5 to 6 2024-04-19T02:03:13.753 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 34 v 6 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-19T02:03:13.753 INFO:tasks.rados.rados.0.smithi059.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-04-19T02:03:13.753 INFO:tasks.rados.rados.0.smithi059.stdout:33: done (3 left) 2024-04-19T02:03:13.753 INFO:tasks.rados.rados.0.smithi059.stdout:34: done (2 left) 2024-04-19T02:03:13.753 INFO:tasks.rados.rados.0.smithi059.stdout:37: write initial oid 37 2024-04-19T02:03:13.753 INFO:tasks.rados.rados.0.smithi059.stdout:37: seq_num 36 ranges {0=606208} 2024-04-19T02:03:13.757 INFO:tasks.rados.rados.0.smithi059.stdout:37: writing smithi05930959-37 from 0 to 606208 tid 1 2024-04-19T02:03:13.758 INFO:tasks.rados.rados.0.smithi059.stdout:35: finishing write tid 1 to smithi05930959-35 2024-04-19T02:03:13.759 INFO:tasks.rados.rados.0.smithi059.stdout:35: finishing write tid 2 to smithi05930959-35 2024-04-19T02:03:13.759 INFO:tasks.rados.rados.0.smithi059.stdout:35: finishing write tid 3 to smithi05930959-35 2024-04-19T02:03:13.759 INFO:tasks.rados.rados.0.smithi059.stdout:35: oid 35 updating version 0 to 4 2024-04-19T02:03:13.759 INFO:tasks.rados.rados.0.smithi059.stdout:35: oid 35 version 4 is already newer than 3 2024-04-19T02:03:13.759 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 35 v 4 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-19T02:03:13.759 INFO:tasks.rados.rados.0.smithi059.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-04-19T02:03:13.759 INFO:tasks.rados.rados.0.smithi059.stdout:35: done (2 left) 2024-04-19T02:03:13.759 INFO:tasks.rados.rados.0.smithi059.stdout:38: write initial oid 38 2024-04-19T02:03:13.759 INFO:tasks.rados.rados.0.smithi059.stdout:38: seq_num 37 ranges {0=622592} 2024-04-19T02:03:13.762 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:13.759+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:13.765 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-19T02:03:13.765 INFO:tasks.rados.rados.0.smithi059.stdout:38: writing smithi05930959-38 from 0 to 622592 tid 1 2024-04-19T02:03:13.766 INFO:tasks.rados.rados.0.smithi059.stdout:36: finishing write tid 1 to smithi05930959-36 2024-04-19T02:03:13.766 INFO:tasks.rados.rados.0.smithi059.stdout:36: finishing write tid 2 to smithi05930959-36 2024-04-19T02:03:13.766 INFO:tasks.rados.rados.0.smithi059.stdout:37: finishing write tid 1 to smithi05930959-37 2024-04-19T02:03:13.766 INFO:tasks.rados.rados.0.smithi059.stdout:39: write initial oid 39 2024-04-19T02:03:13.766 INFO:tasks.rados.rados.0.smithi059.stdout:39: seq_num 38 ranges {0=638976} 2024-04-19T02:03:13.770 INFO:tasks.rados.rados.0.smithi059.stdout:39: writing smithi05930959-39 from 0 to 638976 tid 1 2024-04-19T02:03:13.770 INFO:tasks.rados.rados.0.smithi059.stdout:40: write initial oid 40 2024-04-19T02:03:13.770 INFO:tasks.rados.rados.0.smithi059.stdout:40: seq_num 39 ranges {0=671744} 2024-04-19T02:03:13.775 INFO:tasks.rados.rados.0.smithi059.stdout:40: writing smithi05930959-40 from 0 to 671744 tid 1 2024-04-19T02:03:13.776 INFO:tasks.rados.rados.0.smithi059.stdout:41: write initial oid 41 2024-04-19T02:03:13.776 INFO:tasks.rados.rados.0.smithi059.stdout:41: seq_num 40 ranges {0=688128} 2024-04-19T02:03:13.781 INFO:tasks.rados.rados.0.smithi059.stdout:41: writing smithi05930959-41 from 0 to 688128 tid 1 2024-04-19T02:03:13.781 INFO:tasks.rados.rados.0.smithi059.stdout:42: write initial oid 42 2024-04-19T02:03:13.782 INFO:tasks.rados.rados.0.smithi059.stdout:42: seq_num 41 ranges {0=704512} 2024-04-19T02:03:13.787 INFO:tasks.rados.rados.0.smithi059.stdout:42: writing smithi05930959-42 from 0 to 704512 tid 1 2024-04-19T02:03:13.788 INFO:tasks.rados.rados.0.smithi059.stdout:43: write initial oid 43 2024-04-19T02:03:13.788 INFO:tasks.rados.rados.0.smithi059.stdout:43: seq_num 42 ranges {0=720896} 2024-04-19T02:03:13.793 INFO:tasks.rados.rados.0.smithi059.stdout:43: writing smithi05930959-43 from 0 to 720896 tid 1 2024-04-19T02:03:13.794 INFO:tasks.rados.rados.0.smithi059.stdout:44: write initial oid 44 2024-04-19T02:03:13.794 INFO:tasks.rados.rados.0.smithi059.stdout:44: seq_num 43 ranges {0=737280} 2024-04-19T02:03:13.801 INFO:tasks.rados.rados.0.smithi059.stdout:44: writing smithi05930959-44 from 0 to 737280 tid 1 2024-04-19T02:03:13.801 INFO:tasks.rados.rados.0.smithi059.stdout:45: write initial oid 45 2024-04-19T02:03:13.801 INFO:tasks.rados.rados.0.smithi059.stdout:45: seq_num 44 ranges {0=753664} 2024-04-19T02:03:13.807 INFO:tasks.rados.rados.0.smithi059.stdout:45: writing smithi05930959-45 from 0 to 753664 tid 1 2024-04-19T02:03:13.807 INFO:tasks.rados.rados.0.smithi059.stdout:46: write initial oid 46 2024-04-19T02:03:13.807 INFO:tasks.rados.rados.0.smithi059.stdout:46: seq_num 45 ranges {0=770048} 2024-04-19T02:03:13.812 INFO:tasks.rados.rados.0.smithi059.stdout:46: writing smithi05930959-46 from 0 to 770048 tid 1 2024-04-19T02:03:13.813 INFO:tasks.rados.rados.0.smithi059.stdout:47: write initial oid 47 2024-04-19T02:03:13.813 INFO:tasks.rados.rados.0.smithi059.stdout:47: seq_num 46 ranges {0=786432} 2024-04-19T02:03:13.819 INFO:tasks.rados.rados.0.smithi059.stdout:47: writing smithi05930959-47 from 0 to 786432 tid 1 2024-04-19T02:03:13.820 INFO:tasks.rados.rados.0.smithi059.stdout:48: write initial oid 48 2024-04-19T02:03:13.820 INFO:tasks.rados.rados.0.smithi059.stdout:48: seq_num 47 ranges {0=425984,425984=376832} 2024-04-19T02:03:13.824 INFO:tasks.rados.rados.0.smithi059.stdout:48: writing smithi05930959-48 from 0 to 425984 tid 1 2024-04-19T02:03:13.829 INFO:tasks.rados.rados.0.smithi059.stdout:48: writing smithi05930959-48 from 425984 to 802816 tid 2 2024-04-19T02:03:13.829 INFO:tasks.rados.rados.0.smithi059.stdout:36: finishing write tid 3 to smithi05930959-36 2024-04-19T02:03:13.829 INFO:tasks.rados.rados.0.smithi059.stdout:36: oid 36 updating version 0 to 5 2024-04-19T02:03:13.829 INFO:tasks.rados.rados.0.smithi059.stdout:36: oid 36 updating version 5 to 6 2024-04-19T02:03:13.829 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 36 v 6 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-19T02:03:13.829 INFO:tasks.rados.rados.0.smithi059.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-04-19T02:03:13.829 INFO:tasks.rados.rados.0.smithi059.stdout:37: finishing write tid 2 to smithi05930959-37 2024-04-19T02:03:13.829 INFO:tasks.rados.rados.0.smithi059.stdout:37: finishing write tid 3 to smithi05930959-37 2024-04-19T02:03:13.829 INFO:tasks.rados.rados.0.smithi059.stdout:37: oid 37 updating version 0 to 6 2024-04-19T02:03:13.829 INFO:tasks.rados.rados.0.smithi059.stdout:37: oid 37 version 6 is already newer than 5 2024-04-19T02:03:13.829 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 37 v 6 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-19T02:03:13.829 INFO:tasks.rados.rados.0.smithi059.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-04-19T02:03:13.829 INFO:tasks.rados.rados.0.smithi059.stdout:38: finishing write tid 1 to smithi05930959-38 2024-04-19T02:03:13.830 INFO:tasks.rados.rados.0.smithi059.stdout:38: finishing write tid 2 to smithi05930959-38 2024-04-19T02:03:13.830 INFO:tasks.rados.rados.0.smithi059.stdout:39: finishing write tid 1 to smithi05930959-39 2024-04-19T02:03:13.830 INFO:tasks.rados.rados.0.smithi059.stdout:38: finishing write tid 3 to smithi05930959-38 2024-04-19T02:03:13.830 INFO:tasks.rados.rados.0.smithi059.stdout:38: oid 38 updating version 0 to 8 2024-04-19T02:03:13.830 INFO:tasks.rados.rados.0.smithi059.stdout:38: oid 38 version 8 is already newer than 7 2024-04-19T02:03:13.830 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 38 v 8 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-19T02:03:13.830 INFO:tasks.rados.rados.0.smithi059.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-04-19T02:03:13.830 INFO:tasks.rados.rados.0.smithi059.stdout:39: finishing write tid 2 to smithi05930959-39 2024-04-19T02:03:13.830 INFO:tasks.rados.rados.0.smithi059.stdout:39: finishing write tid 3 to smithi05930959-39 2024-04-19T02:03:13.830 INFO:tasks.rados.rados.0.smithi059.stdout:39: oid 39 updating version 0 to 8 2024-04-19T02:03:13.830 INFO:tasks.rados.rados.0.smithi059.stdout:39: oid 39 version 8 is already newer than 7 2024-04-19T02:03:13.830 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 39 v 8 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-19T02:03:13.830 INFO:tasks.rados.rados.0.smithi059.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-04-19T02:03:13.830 INFO:tasks.rados.rados.0.smithi059.stdout:40: finishing write tid 1 to smithi05930959-40 2024-04-19T02:03:13.830 INFO:tasks.rados.rados.0.smithi059.stdout:36: done (12 left) 2024-04-19T02:03:13.830 INFO:tasks.rados.rados.0.smithi059.stdout:37: done (11 left) 2024-04-19T02:03:13.831 INFO:tasks.rados.rados.0.smithi059.stdout:38: done (10 left) 2024-04-19T02:03:13.831 INFO:tasks.rados.rados.0.smithi059.stdout:39: done (9 left) 2024-04-19T02:03:13.831 INFO:tasks.rados.rados.0.smithi059.stdout:49: write initial oid 49 2024-04-19T02:03:13.831 INFO:tasks.rados.rados.0.smithi059.stdout:49: seq_num 48 ranges {0=442368,442368=376832} 2024-04-19T02:03:13.833 INFO:tasks.rados.rados.0.smithi059.stdout:49: writing smithi05930959-49 from 0 to 442368 tid 1 2024-04-19T02:03:13.837 INFO:tasks.rados.rados.0.smithi059.stdout:49: writing smithi05930959-49 from 442368 to 819200 tid 2 2024-04-19T02:03:13.837 INFO:tasks.rados.rados.0.smithi059.stdout:50: write initial oid 50 2024-04-19T02:03:13.837 INFO:tasks.rados.rados.0.smithi059.stdout:50: seq_num 49 ranges {0=458752,458752=376832} 2024-04-19T02:03:13.840 INFO:tasks.rados.rados.0.smithi059.stdout:50: writing smithi05930959-50 from 0 to 458752 tid 1 2024-04-19T02:03:13.844 INFO:tasks.rados.rados.0.smithi059.stdout:50: writing smithi05930959-50 from 458752 to 835584 tid 2 2024-04-19T02:03:13.844 INFO:tasks.rados.rados.0.smithi059.stdout:51: setattr oid 20 current snap is 0 2024-04-19T02:03:13.844 INFO:tasks.rados.rados.0.smithi059.stdout:40: finishing write tid 2 to smithi05930959-40 2024-04-19T02:03:13.844 INFO:tasks.rados.rados.0.smithi059.stdout:40: finishing write tid 3 to smithi05930959-40 2024-04-19T02:03:13.844 INFO:tasks.rados.rados.0.smithi059.stdout:40: oid 40 updating version 0 to 7 2024-04-19T02:03:13.844 INFO:tasks.rados.rados.0.smithi059.stdout:40: oid 40 updating version 7 to 8 2024-04-19T02:03:13.844 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 40 v 8 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-19T02:03:13.844 INFO:tasks.rados.rados.0.smithi059.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-04-19T02:03:13.844 INFO:tasks.rados.rados.0.smithi059.stdout:41: finishing write tid 1 to smithi05930959-41 2024-04-19T02:03:13.844 INFO:tasks.rados.rados.0.smithi059.stdout:41: finishing write tid 2 to smithi05930959-41 2024-04-19T02:03:13.844 INFO:tasks.rados.rados.0.smithi059.stdout:41: finishing write tid 3 to smithi05930959-41 2024-04-19T02:03:13.844 INFO:tasks.rados.rados.0.smithi059.stdout:41: oid 41 updating version 0 to 6 2024-04-19T02:03:13.844 INFO:tasks.rados.rados.0.smithi059.stdout:41: oid 41 version 6 is already newer than 5 2024-04-19T02:03:13.844 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 41 v 6 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-19T02:03:13.845 INFO:tasks.rados.rados.0.smithi059.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-04-19T02:03:13.845 INFO:tasks.rados.rados.0.smithi059.stdout:42: finishing write tid 1 to smithi05930959-42 2024-04-19T02:03:13.845 INFO:tasks.rados.rados.0.smithi059.stdout:42: finishing write tid 2 to smithi05930959-42 2024-04-19T02:03:13.845 INFO:tasks.rados.rados.0.smithi059.stdout:42: finishing write tid 3 to smithi05930959-42 2024-04-19T02:03:13.845 INFO:tasks.rados.rados.0.smithi059.stdout:42: oid 42 updating version 0 to 6 2024-04-19T02:03:13.845 INFO:tasks.rados.rados.0.smithi059.stdout:42: oid 42 version 6 is already newer than 5 2024-04-19T02:03:13.845 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 42 v 6 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-19T02:03:13.845 INFO:tasks.rados.rados.0.smithi059.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-04-19T02:03:13.846 INFO:tasks.rados.rados.0.smithi059.stdout:43: finishing write tid 1 to smithi05930959-43 2024-04-19T02:03:13.846 INFO:tasks.rados.rados.0.smithi059.stdout:43: finishing write tid 2 to smithi05930959-43 2024-04-19T02:03:13.846 INFO:tasks.rados.rados.0.smithi059.stdout:43: finishing write tid 3 to smithi05930959-43 2024-04-19T02:03:13.846 INFO:tasks.rados.rados.0.smithi059.stdout:43: oid 43 updating version 0 to 3 2024-04-19T02:03:13.846 INFO:tasks.rados.rados.0.smithi059.stdout:43: oid 43 updating version 3 to 4 2024-04-19T02:03:13.846 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 43 v 4 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-19T02:03:13.846 INFO:tasks.rados.rados.0.smithi059.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-04-19T02:03:13.846 INFO:tasks.rados.rados.0.smithi059.stdout:44: finishing write tid 1 to smithi05930959-44 2024-04-19T02:03:13.846 INFO:tasks.rados.rados.0.smithi059.stdout:44: finishing write tid 2 to smithi05930959-44 2024-04-19T02:03:13.846 INFO:tasks.rados.rados.0.smithi059.stdout:44: finishing write tid 3 to smithi05930959-44 2024-04-19T02:03:13.847 INFO:tasks.rados.rados.0.smithi059.stdout:44: oid 44 updating version 0 to 10 2024-04-19T02:03:13.847 INFO:tasks.rados.rados.0.smithi059.stdout:44: oid 44 version 10 is already newer than 9 2024-04-19T02:03:13.847 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 44 v 10 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-19T02:03:13.847 INFO:tasks.rados.rados.0.smithi059.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-04-19T02:03:13.847 INFO:tasks.rados.rados.0.smithi059.stdout:45: finishing write tid 1 to smithi05930959-45 2024-04-19T02:03:13.847 INFO:tasks.rados.rados.0.smithi059.stdout:45: finishing write tid 2 to smithi05930959-45 2024-04-19T02:03:13.847 INFO:tasks.rados.rados.0.smithi059.stdout:45: finishing write tid 3 to smithi05930959-45 2024-04-19T02:03:13.847 INFO:tasks.rados.rados.0.smithi059.stdout:45: oid 45 updating version 0 to 6 2024-04-19T02:03:13.847 INFO:tasks.rados.rados.0.smithi059.stdout:45: oid 45 version 6 is already newer than 5 2024-04-19T02:03:13.847 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 45 v 6 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-19T02:03:13.847 INFO:tasks.rados.rados.0.smithi059.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-04-19T02:03:13.847 INFO:tasks.rados.rados.0.smithi059.stdout:46: finishing write tid 1 to smithi05930959-46 2024-04-19T02:03:13.847 INFO:tasks.rados.rados.0.smithi059.stdout:46: finishing write tid 2 to smithi05930959-46 2024-04-19T02:03:13.847 INFO:tasks.rados.rados.0.smithi059.stdout:46: finishing write tid 3 to smithi05930959-46 2024-04-19T02:03:13.847 INFO:tasks.rados.rados.0.smithi059.stdout:46: oid 46 updating version 0 to 3 2024-04-19T02:03:13.848 INFO:tasks.rados.rados.0.smithi059.stdout:46: oid 46 updating version 3 to 4 2024-04-19T02:03:13.848 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 46 v 4 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-19T02:03:13.848 INFO:tasks.rados.rados.0.smithi059.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-04-19T02:03:13.848 INFO:tasks.rados.rados.0.smithi059.stdout:47: finishing write tid 1 to smithi05930959-47 2024-04-19T02:03:13.848 INFO:tasks.rados.rados.0.smithi059.stdout:47: finishing write tid 2 to smithi05930959-47 2024-04-19T02:03:13.848 INFO:tasks.rados.rados.0.smithi059.stdout:47: finishing write tid 3 to smithi05930959-47 2024-04-19T02:03:13.848 INFO:tasks.rados.rados.0.smithi059.stdout:47: oid 47 updating version 0 to 5 2024-04-19T02:03:13.848 INFO:tasks.rados.rados.0.smithi059.stdout:47: oid 47 updating version 5 to 6 2024-04-19T02:03:13.848 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 47 v 6 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-19T02:03:13.848 INFO:tasks.rados.rados.0.smithi059.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-04-19T02:03:13.848 INFO:tasks.rados.rados.0.smithi059.stdout:48: finishing write tid 1 to smithi05930959-48 2024-04-19T02:03:13.848 INFO:tasks.rados.rados.0.smithi059.stdout:48: finishing write tid 2 to smithi05930959-48 2024-04-19T02:03:13.848 INFO:tasks.rados.rados.0.smithi059.stdout:48: finishing write tid 3 to smithi05930959-48 2024-04-19T02:03:13.848 INFO:tasks.rados.rados.0.smithi059.stdout:48: finishing write tid 4 to smithi05930959-48 2024-04-19T02:03:13.848 INFO:tasks.rados.rados.0.smithi059.stdout:48: oid 48 updating version 0 to 8 2024-04-19T02:03:13.848 INFO:tasks.rados.rados.0.smithi059.stdout:48: oid 48 updating version 8 to 9 2024-04-19T02:03:13.849 INFO:tasks.rados.rados.0.smithi059.stdout:48: oid 48 version 9 is already newer than 7 2024-04-19T02:03:13.849 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 48 v 9 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-19T02:03:13.849 INFO:tasks.rados.rados.0.smithi059.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-04-19T02:03:13.849 INFO:tasks.rados.rados.0.smithi059.stdout:49: finishing write tid 1 to smithi05930959-49 2024-04-19T02:03:13.850 INFO:tasks.rados.rados.0.smithi059.stdout:49: finishing write tid 2 to smithi05930959-49 2024-04-19T02:03:13.851 INFO:tasks.rados.rados.0.smithi059.stdout:49: finishing write tid 3 to smithi05930959-49 2024-04-19T02:03:13.851 INFO:tasks.rados.rados.0.smithi059.stdout:40: done (11 left) 2024-04-19T02:03:13.851 INFO:tasks.rados.rados.0.smithi059.stdout:41: done (10 left) 2024-04-19T02:03:13.851 INFO:tasks.rados.rados.0.smithi059.stdout:42: done (9 left) 2024-04-19T02:03:13.851 INFO:tasks.rados.rados.0.smithi059.stdout:43: done (8 left) 2024-04-19T02:03:13.851 INFO:tasks.rados.rados.0.smithi059.stdout:44: done (7 left) 2024-04-19T02:03:13.851 INFO:tasks.rados.rados.0.smithi059.stdout:45: done (6 left) 2024-04-19T02:03:13.851 INFO:tasks.rados.rados.0.smithi059.stdout:46: done (5 left) 2024-04-19T02:03:13.851 INFO:tasks.rados.rados.0.smithi059.stdout:47: done (4 left) 2024-04-19T02:03:13.851 INFO:tasks.rados.rados.0.smithi059.stdout:48: done (3 left) 2024-04-19T02:03:13.851 INFO:tasks.rados.rados.0.smithi059.stdout:52: rollback oid 34 current snap is 0 2024-04-19T02:03:13.851 INFO:tasks.rados.rados.0.smithi059.stdout:52: done (3 left) 2024-04-19T02:03:13.851 INFO:tasks.rados.rados.0.smithi059.stdout:53: rollback oid 1 current snap is 0 2024-04-19T02:03:13.851 INFO:tasks.rados.rados.0.smithi059.stdout:53: done (3 left) 2024-04-19T02:03:13.851 INFO:tasks.rados.rados.0.smithi059.stdout:54: rmattr oid 24 current snap is 0 2024-04-19T02:03:13.852 INFO:tasks.rados.rados.0.smithi059.stdout:54: done (3 left) 2024-04-19T02:03:13.852 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 47 current snap is 0 2024-04-19T02:03:13.852 INFO:tasks.rados.rados.0.smithi059.stdout:55: seq_num 50 ranges {786432=475136,1261568=376832} 2024-04-19T02:03:13.853 INFO:tasks.rados.rados.0.smithi059.stdout:55: writing smithi05930959-47 from 786432 to 1261568 tid 1 2024-04-19T02:03:13.856 INFO:tasks.rados.rados.0.smithi059.stdout:55: writing smithi05930959-47 from 1261568 to 1638400 tid 2 2024-04-19T02:03:13.857 INFO:tasks.rados.rados.0.smithi059.stdout:50: finishing write tid 1 to smithi05930959-50 2024-04-19T02:03:13.857 INFO:tasks.rados.rados.0.smithi059.stdout:49: finishing write tid 4 to smithi05930959-49 2024-04-19T02:03:13.857 INFO:tasks.rados.rados.0.smithi059.stdout:49: oid 49 updating version 0 to 6 2024-04-19T02:03:13.857 INFO:tasks.rados.rados.0.smithi059.stdout:49: oid 49 updating version 6 to 7 2024-04-19T02:03:13.857 INFO:tasks.rados.rados.0.smithi059.stdout:49: oid 49 version 7 is already newer than 5 2024-04-19T02:03:13.857 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 49 v 7 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-19T02:03:13.857 INFO:tasks.rados.rados.0.smithi059.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-04-19T02:03:13.857 INFO:tasks.rados.rados.0.smithi059.stdout:50: finishing write tid 2 to smithi05930959-50 2024-04-19T02:03:13.857 INFO:tasks.rados.rados.0.smithi059.stdout:50: finishing write tid 3 to smithi05930959-50 2024-04-19T02:03:13.857 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 20 v 10 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-19T02:03:13.857 INFO:tasks.rados.rados.0.smithi059.stdout:50: finishing write tid 4 to smithi05930959-50 2024-04-19T02:03:13.857 INFO:tasks.rados.rados.0.smithi059.stdout:50: oid 50 updating version 0 to 7 2024-04-19T02:03:13.857 INFO:tasks.rados.rados.0.smithi059.stdout:50: oid 50 updating version 7 to 8 2024-04-19T02:03:13.858 INFO:tasks.rados.rados.0.smithi059.stdout:50: oid 50 updating version 8 to 9 2024-04-19T02:03:13.858 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 50 v 9 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-19T02:03:13.858 INFO:tasks.rados.rados.0.smithi059.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-04-19T02:03:13.858 INFO:tasks.rados.rados.0.smithi059.stdout:49: done (3 left) 2024-04-19T02:03:13.858 INFO:tasks.rados.rados.0.smithi059.stdout:50: done (2 left) 2024-04-19T02:03:13.858 INFO:tasks.rados.rados.0.smithi059.stdout:51: done (1 left) 2024-04-19T02:03:13.858 INFO:tasks.rados.rados.0.smithi059.stdout:56: delete oid 12 current snap is 0 2024-04-19T02:03:13.862 INFO:tasks.rados.rados.0.smithi059.stdout:55: finishing write tid 1 to smithi05930959-47 2024-04-19T02:03:13.862 INFO:tasks.rados.rados.0.smithi059.stdout:55: finishing write tid 2 to smithi05930959-47 2024-04-19T02:03:13.862 INFO:tasks.rados.rados.0.smithi059.stdout:55: finishing write tid 3 to smithi05930959-47 2024-04-19T02:03:13.862 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:13.863+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:13.863 INFO:tasks.rados.rados.0.smithi059.stdout:56: done (1 left) 2024-04-19T02:03:13.863 INFO:tasks.rados.rados.0.smithi059.stdout:57: read oid 9 snap -1 2024-04-19T02:03:13.863 INFO:tasks.rados.rados.0.smithi059.stdout:57: expect (ObjNum 8 snap 0 seq_num 8) 2024-04-19T02:03:13.863 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 25 current snap is 0 2024-04-19T02:03:13.863 INFO:tasks.rados.rados.0.smithi059.stdout:58: seq_num 52 ranges {409600=507904,917504=376832} 2024-04-19T02:03:13.868 INFO:tasks.rados.rados.0.smithi059.stdout:58: writing smithi05930959-25 from 409600 to 917504 tid 1 2024-04-19T02:03:13.871 INFO:tasks.rados.rados.0.smithi059.stdout:58: writing smithi05930959-25 from 917504 to 1294336 tid 2 2024-04-19T02:03:13.871 INFO:tasks.rados.rados.0.smithi059.stdout:59: read oid 28 snap -1 2024-04-19T02:03:13.871 INFO:tasks.rados.rados.0.smithi059.stdout:59: expect (ObjNum 27 snap 0 seq_num 27) 2024-04-19T02:03:13.871 INFO:tasks.rados.rados.0.smithi059.stdout:60: read oid 10 snap -1 2024-04-19T02:03:13.871 INFO:tasks.rados.rados.0.smithi059.stdout:60: expect (ObjNum 9 snap 0 seq_num 9) 2024-04-19T02:03:13.871 INFO:tasks.rados.rados.0.smithi059.stdout:61: rollback oid 21 current snap is 0 2024-04-19T02:03:13.871 INFO:tasks.rados.rados.0.smithi059.stdout:61: done (5 left) 2024-04-19T02:03:13.872 INFO:tasks.rados.rados.0.smithi059.stdout:62: read oid 35 snap -1 2024-04-19T02:03:13.872 INFO:tasks.rados.rados.0.smithi059.stdout:62: expect (ObjNum 34 snap 0 seq_num 34) 2024-04-19T02:03:13.872 INFO:tasks.rados.rados.0.smithi059.stdout:63: delete oid 17 current snap is 0 2024-04-19T02:03:13.872 INFO:tasks.rados.rados.0.smithi059.stdout:55: finishing write tid 4 to smithi05930959-47 2024-04-19T02:03:13.872 INFO:tasks.rados.rados.0.smithi059.stdout:55: oid 47 updating version 0 to 9 2024-04-19T02:03:13.872 INFO:tasks.rados.rados.0.smithi059.stdout:55: oid 47 version 9 is already newer than 7 2024-04-19T02:03:13.872 INFO:tasks.rados.rados.0.smithi059.stdout:55: oid 47 version 9 is already newer than 8 2024-04-19T02:03:13.872 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 47 v 9 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-04-19T02:03:13.872 INFO:tasks.rados.rados.0.smithi059.stdout:55: left oid 47 (ObjNum 50 snap 0 seq_num 50) 2024-04-19T02:03:13.872 INFO:tasks.rados.rados.0.smithi059.stdout:57: expect (ObjNum 8 snap 0 seq_num 8) 2024-04-19T02:03:13.879 INFO:tasks.rados.rados.0.smithi059.stdout:58: finishing write tid 1 to smithi05930959-25 2024-04-19T02:03:13.879 INFO:tasks.rados.rados.0.smithi059.stdout:62: expect (ObjNum 34 snap 0 seq_num 34) 2024-04-19T02:03:13.895 INFO:tasks.rados.rados.0.smithi059.stdout:60: expect (ObjNum 9 snap 0 seq_num 9) 2024-04-19T02:03:13.899 INFO:tasks.rados.rados.0.smithi059.stdout:59: expect (ObjNum 27 snap 0 seq_num 27) 2024-04-19T02:03:13.912 INFO:tasks.rados.rados.0.smithi059.stdout:58: finishing write tid 2 to smithi05930959-25 2024-04-19T02:03:13.913 INFO:tasks.rados.rados.0.smithi059.stdout:58: finishing write tid 3 to smithi05930959-25 2024-04-19T02:03:13.913 INFO:tasks.rados.rados.0.smithi059.stdout:58: finishing write tid 4 to smithi05930959-25 2024-04-19T02:03:13.913 INFO:tasks.rados.rados.0.smithi059.stdout:58: oid 25 updating version 0 to 10 2024-04-19T02:03:13.913 INFO:tasks.rados.rados.0.smithi059.stdout:58: oid 25 updating version 10 to 12 2024-04-19T02:03:13.913 INFO:tasks.rados.rados.0.smithi059.stdout:58: oid 25 version 12 is already newer than 11 2024-04-19T02:03:13.913 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 25 v 12 (ObjNum 52 snap 0 seq_num 52) dirty exists 2024-04-19T02:03:13.913 INFO:tasks.rados.rados.0.smithi059.stdout:58: left oid 25 (ObjNum 52 snap 0 seq_num 52) 2024-04-19T02:03:13.913 INFO:tasks.rados.rados.0.smithi059.stdout:55: done (6 left) 2024-04-19T02:03:13.913 INFO:tasks.rados.rados.0.smithi059.stdout:57: done (5 left) 2024-04-19T02:03:13.913 INFO:tasks.rados.rados.0.smithi059.stdout:58: done (4 left) 2024-04-19T02:03:13.913 INFO:tasks.rados.rados.0.smithi059.stdout:59: done (3 left) 2024-04-19T02:03:13.913 INFO:tasks.rados.rados.0.smithi059.stdout:60: done (2 left) 2024-04-19T02:03:13.913 INFO:tasks.rados.rados.0.smithi059.stdout:62: done (1 left) 2024-04-19T02:03:13.914 INFO:tasks.rados.rados.0.smithi059.stdout:63: done (0 left) 2024-04-19T02:03:13.914 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 27 current snap is 0 2024-04-19T02:03:13.914 INFO:tasks.rados.rados.0.smithi059.stdout:64: seq_num 54 ranges {442368=540672,983040=376832} 2024-04-19T02:03:13.919 INFO:tasks.rados.rados.0.smithi059.stdout:64: writing smithi05930959-27 from 442368 to 983040 tid 1 2024-04-19T02:03:13.922 INFO:tasks.rados.rados.0.smithi059.stdout:64: writing smithi05930959-27 from 983040 to 1359872 tid 2 2024-04-19T02:03:13.922 INFO:tasks.rados.rados.0.smithi059.stdout:65: delete oid 45 current snap is 0 2024-04-19T02:03:13.926 INFO:tasks.rados.rados.0.smithi059.stdout:64: finishing write tid 1 to smithi05930959-27 2024-04-19T02:03:13.929 INFO:tasks.rados.rados.0.smithi059.stdout:65: done (1 left) 2024-04-19T02:03:13.929 INFO:tasks.rados.rados.0.smithi059.stdout:66: snap_create 2024-04-19T02:03:13.929 INFO:tasks.rados.rados.0.smithi059.stdout:64: finishing write tid 2 to smithi05930959-27 2024-04-19T02:03:13.929 INFO:tasks.rados.rados.0.smithi059.stdout:64: finishing write tid 3 to smithi05930959-27 2024-04-19T02:03:13.931 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:13.931 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:13.931 INFO:tasks.rados.rados.0.smithi059.stdout:64: finishing write tid 4 to smithi05930959-27 2024-04-19T02:03:13.931 INFO:tasks.rados.rados.0.smithi059.stdout:64: oid 27 updating version 0 to 18 2024-04-19T02:03:13.931 INFO:tasks.rados.rados.0.smithi059.stdout:64: oid 27 version 18 is already newer than 17 2024-04-19T02:03:13.931 INFO:tasks.rados.rados.0.smithi059.stdout:64: oid 27 version 18 is already newer than 16 2024-04-19T02:03:13.931 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 27 v 18 (ObjNum 54 snap 0 seq_num 54) dirty exists 2024-04-19T02:03:13.932 INFO:tasks.rados.rados.0.smithi059.stdout:64: left oid 27 (ObjNum 54 snap 0 seq_num 54) 2024-04-19T02:03:13.945 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-19T02:03:13.963 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:13.963+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:14.007 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:03:14.008 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-19T02:03:14.025 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-04-19T02:03:13.616639+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":52,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":11,"kb":1132462080,"kb_used":328136,"kb_used_data":3860,"kb_used_omap":18,"kb_used_meta":324077,"kb_avail":1132133944,"statfs":{"total":1159641169920,"available":1159305158656,"internally_reserved":0,"allocated":3952640,"data_stored":2239969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":331855161},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.030300"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-19T02:03:10.585869+0000","last_change":"2024-04-19T02:03:10.585869+0000","last_active":"2024-04-19T02:03:10.585869+0000","last_peered":"2024-04-19T02:03:10.585869+0000","last_clean":"2024-04-19T02:03:10.585869+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-19T02:03:10.585869+0000","last_undegraded":"2024-04-19T02:03:10.585869+0000","last_fullsized":"2024-04-19T02:03:10.585869+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-19T02:03:10.585869+0000","last_change":"2024-04-19T02:03:10.585869+0000","last_active":"2024-04-19T02:03:10.585869+0000","last_peered":"2024-04-19T02:03:10.585869+0000","last_clean":"2024-04-19T02:03:10.585869+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-19T02:03:10.585869+0000","last_undegraded":"2024-04-19T02:03:10.585869+0000","last_fullsized":"2024-04-19T02:03:10.585869+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-19T02:03:10.585869+0000","last_change":"2024-04-19T02:03:10.585869+0000","last_active":"2024-04-19T02:03:10.585869+0000","last_peered":"2024-04-19T02:03:10.585869+0000","last_clean":"2024-04-19T02:03:10.585869+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-19T02:03:10.585869+0000","last_undegraded":"2024-04-19T02:03:10.585869+0000","last_fullsized":"2024-04-19T02:03:10.585869+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-19T02:03:10.585869+0000","last_change":"2024-04-19T02:03:10.585869+0000","last_active":"2024-04-19T02:03:10.585869+0000","last_peered":"2024-04-19T02:03:10.585869+0000","last_clean":"2024-04-19T02:03:10.585869+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-19T02:03:10.585869+0000","last_undegraded":"2024-04-19T02:03:10.585869+0000","last_fullsized":"2024-04-19T02:03:10.585869+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-19T02:03:10.585869+0000","last_change":"2024-04-19T02:03:10.585869+0000","last_active":"2024-04-19T02:03:10.585869+0000","last_peered":"2024-04-19T02:03:10.585869+0000","last_clean":"2024-04-19T02:03:10.585869+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-19T02:03:10.585869+0000","last_undegraded":"2024-04-19T02:03:10.585869+0000","last_fullsized":"2024-04-19T02:03:10.585869+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-19T02:03:10.585869+0000","last_change":"2024-04-19T02:03:10.585869+0000","last_active":"2024-04-19T02:03:10.585869+0000","last_peered":"2024-04-19T02:03:10.585869+0000","last_clean":"2024-04-19T02:03:10.585869+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-19T02:03:10.585869+0000","last_undegraded":"2024-04-19T02:03:10.585869+0000","last_fullsized":"2024-04-19T02:03:10.585869+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-19T02:03:10.585869+0000","last_change":"2024-04-19T02:03:10.585869+0000","last_active":"2024-04-19T02:03:10.585869+0000","last_peered":"2024-04-19T02:03:10.585869+0000","last_clean":"2024-04-19T02:03:10.585869+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-19T02:03:10.585869+0000","last_undegraded":"2024-04-19T02:03:10.585869+0000","last_fullsized":"2024-04-19T02:03:10.585869+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-19T02:03:10.585869+0000","last_change":"2024-04-19T02:03:10.585869+0000","last_active":"2024-04-19T02:03:10.585869+0000","last_peered":"2024-04-19T02:03:10.585869+0000","last_clean":"2024-04-19T02:03:10.585869+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-19T02:03:10.585869+0000","last_undegraded":"2024-04-19T02:03:10.585869+0000","last_fullsized":"2024-04-19T02:03:10.585869+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":193,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-19T02:03:08.586753+0000","last_change":"2024-04-19T02:02:59.974166+0000","last_active":"2024-04-19T02:03:08.586753+0000","last_peered":"2024-04-19T02:03:08.586753+0000","last_clean":"2024-04-19T02:03:08.586753+0000","last_became_active":"2024-04-19T02:02:59.972005+0000","last_became_peered":"2024-04-19T02:02:59.972005+0000","last_unstale":"2024-04-19T02:03:08.586753+0000","last_undegraded":"2024-04-19T02:03:08.586753+0000","last_fullsized":"2024-04-19T02:03:08.586753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:02:58.486225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:02:58.486225+0000","last_clean_scrub_stamp":"2024-04-19T02:02:58.486225+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:20.275707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-19T02:03:08.741304+0000","last_change":"2024-04-19T02:03:03.026806+0000","last_active":"2024-04-19T02:03:08.741304+0000","last_peered":"2024-04-19T02:03:08.741304+0000","last_clean":"2024-04-19T02:03:08.741304+0000","last_became_active":"2024-04-19T02:03:03.026063+0000","last_became_peered":"2024-04-19T02:03:03.026063+0000","last_unstale":"2024-04-19T02:03:08.741304+0000","last_undegraded":"2024-04-19T02:03:08.741304+0000","last_fullsized":"2024-04-19T02:03:08.741304+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:19.673016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-19T02:03:10.585869+0000","last_change":"2024-04-19T02:03:10.585869+0000","last_active":"2024-04-19T02:03:10.585869+0000","last_peered":"2024-04-19T02:03:10.585869+0000","last_clean":"2024-04-19T02:03:10.585869+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-19T02:03:10.585869+0000","last_undegraded":"2024-04-19T02:03:10.585869+0000","last_fullsized":"2024-04-19T02:03:10.585869+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-19T02:03:08.586736+0000","last_change":"2024-04-19T02:03:02.561076+0000","last_active":"2024-04-19T02:03:08.586736+0000","last_peered":"2024-04-19T02:03:08.586736+0000","last_clean":"2024-04-19T02:03:08.586736+0000","last_became_active":"2024-04-19T02:03:02.560210+0000","last_became_peered":"2024-04-19T02:03:02.560210+0000","last_unstale":"2024-04-19T02:03:08.586736+0000","last_undegraded":"2024-04-19T02:03:08.586736+0000","last_fullsized":"2024-04-19T02:03:08.586736+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:16.900004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-19T02:03:10.585869+0000","last_change":"2024-04-19T02:03:10.585869+0000","last_active":"2024-04-19T02:03:10.585869+0000","last_peered":"2024-04-19T02:03:10.585869+0000","last_clean":"2024-04-19T02:03:10.585869+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-19T02:03:10.585869+0000","last_undegraded":"2024-04-19T02:03:10.585869+0000","last_fullsized":"2024-04-19T02:03:10.585869+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3,"reported_epoch":24,"state":"peering","last_fresh":"2024-04-19T02:03:08.732825+0000","last_change":"2024-04-19T02:03:08.732825+0000","last_active":"2024-04-19T02:03:01.531230+0000","last_peered":"2024-04-19T02:03:01.531230+0000","last_clean":"2024-04-19T02:03:01.531230+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-19T02:03:08.732825+0000","last_undegraded":"2024-04-19T02:03:08.732825+0000","last_fullsized":"2024-04-19T02:03:08.732825+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T02:03:12.611925+0000","last_change":"2024-04-19T02:03:11.679548+0000","last_active":"2024-04-19T02:03:12.611925+0000","last_peered":"2024-04-19T02:03:12.611925+0000","last_clean":"2024-04-19T02:03:12.611925+0000","last_became_active":"2024-04-19T02:03:11.678078+0000","last_became_peered":"2024-04-19T02:03:11.678078+0000","last_unstale":"2024-04-19T02:03:12.611925+0000","last_undegraded":"2024-04-19T02:03:12.611925+0000","last_fullsized":"2024-04-19T02:03:12.611925+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:14.098891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,9,1,6,4,0,10],"acting":[5,7,9,1,6,4,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-19T02:03:08.586761+0000","last_change":"2024-04-19T02:03:02.560133+0000","last_active":"2024-04-19T02:03:08.586761+0000","last_peered":"2024-04-19T02:03:08.586761+0000","last_clean":"2024-04-19T02:03:08.586761+0000","last_became_active":"2024-04-19T02:03:02.559377+0000","last_became_peered":"2024-04-19T02:03:02.559377+0000","last_unstale":"2024-04-19T02:03:08.586761+0000","last_undegraded":"2024-04-19T02:03:08.586761+0000","last_fullsized":"2024-04-19T02:03:08.586761+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:30.728400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T02:03:13.394666+0000","last_change":"2024-04-19T02:03:11.679342+0000","last_active":"2024-04-19T02:03:13.394666+0000","last_peered":"2024-04-19T02:03:13.394666+0000","last_clean":"2024-04-19T02:03:13.394666+0000","last_became_active":"2024-04-19T02:03:11.677099+0000","last_became_peered":"2024-04-19T02:03:11.677099+0000","last_unstale":"2024-04-19T02:03:13.394666+0000","last_undegraded":"2024-04-19T02:03:13.394666+0000","last_fullsized":"2024-04-19T02:03:13.394666+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:39.696852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,0,3,4,7,9,6],"acting":[10,8,0,3,4,7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T02:03:13.394532+0000","last_change":"2024-04-19T02:03:11.729603+0000","last_active":"2024-04-19T02:03:13.394532+0000","last_peered":"2024-04-19T02:03:13.394532+0000","last_clean":"2024-04-19T02:03:13.394532+0000","last_became_active":"2024-04-19T02:03:11.728949+0000","last_became_peered":"2024-04-19T02:03:11.728949+0000","last_unstale":"2024-04-19T02:03:13.394532+0000","last_undegraded":"2024-04-19T02:03:13.394532+0000","last_fullsized":"2024-04-19T02:03:13.394532+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:13.131235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2,9,8,1,6,5],"acting":[10,3,2,9,8,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T02:03:13.433870+0000","last_change":"2024-04-19T02:03:02.597582+0000","last_active":"2024-04-19T02:03:13.433870+0000","last_peered":"2024-04-19T02:03:13.433870+0000","last_clean":"2024-04-19T02:03:13.433870+0000","last_became_active":"2024-04-19T02:03:02.597118+0000","last_became_peered":"2024-04-19T02:03:02.597118+0000","last_unstale":"2024-04-19T02:03:13.433870+0000","last_undegraded":"2024-04-19T02:03:13.433870+0000","last_fullsized":"2024-04-19T02:03:13.433870+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:13.683143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T02:03:13.433866+0000","last_change":"2024-04-19T02:03:11.731413+0000","last_active":"2024-04-19T02:03:13.433866+0000","last_peered":"2024-04-19T02:03:13.433866+0000","last_clean":"2024-04-19T02:03:13.433866+0000","last_became_active":"2024-04-19T02:03:11.730165+0000","last_became_peered":"2024-04-19T02:03:11.730165+0000","last_unstale":"2024-04-19T02:03:13.433866+0000","last_undegraded":"2024-04-19T02:03:13.433866+0000","last_fullsized":"2024-04-19T02:03:13.433866+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:33.225771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,10,8,7,2,11,0],"acting":[1,5,10,8,7,2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-19T02:03:08.816937+0000","last_change":"2024-04-19T02:03:02.595929+0000","last_active":"2024-04-19T02:03:08.816937+0000","last_peered":"2024-04-19T02:03:08.816937+0000","last_clean":"2024-04-19T02:03:08.816937+0000","last_became_active":"2024-04-19T02:03:02.595232+0000","last_became_peered":"2024-04-19T02:03:02.595232+0000","last_unstale":"2024-04-19T02:03:08.816937+0000","last_undegraded":"2024-04-19T02:03:08.816937+0000","last_fullsized":"2024-04-19T02:03:08.816937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:04.077370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-19T02:03:10.585869+0000","last_change":"2024-04-19T02:03:10.585869+0000","last_active":"2024-04-19T02:03:10.585869+0000","last_peered":"2024-04-19T02:03:10.585869+0000","last_clean":"2024-04-19T02:03:10.585869+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-19T02:03:10.585869+0000","last_undegraded":"2024-04-19T02:03:10.585869+0000","last_fullsized":"2024-04-19T02:03:10.585869+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-19T02:03:13.433869+0000","last_change":"2024-04-19T02:03:02.988302+0000","last_active":"2024-04-19T02:03:13.433869+0000","last_peered":"2024-04-19T02:03:13.433869+0000","last_clean":"2024-04-19T02:03:13.433869+0000","last_became_active":"2024-04-19T02:03:02.988073+0000","last_became_peered":"2024-04-19T02:03:02.988073+0000","last_unstale":"2024-04-19T02:03:13.433869+0000","last_undegraded":"2024-04-19T02:03:13.433869+0000","last_fullsized":"2024-04-19T02:03:13.433869+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:28.455271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-19T02:03:10.585869+0000","last_change":"2024-04-19T02:03:10.585869+0000","last_active":"2024-04-19T02:03:10.585869+0000","last_peered":"2024-04-19T02:03:10.585869+0000","last_clean":"2024-04-19T02:03:10.585869+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-19T02:03:10.585869+0000","last_undegraded":"2024-04-19T02:03:10.585869+0000","last_fullsized":"2024-04-19T02:03:10.585869+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-19T02:03:08.586852+0000","last_change":"2024-04-19T02:03:02.570839+0000","last_active":"2024-04-19T02:03:08.586852+0000","last_peered":"2024-04-19T02:03:08.586852+0000","last_clean":"2024-04-19T02:03:08.586852+0000","last_became_active":"2024-04-19T02:03:02.570116+0000","last_became_peered":"2024-04-19T02:03:02.570116+0000","last_unstale":"2024-04-19T02:03:08.586852+0000","last_undegraded":"2024-04-19T02:03:08.586852+0000","last_fullsized":"2024-04-19T02:03:08.586852+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:23.134425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":19,"seq":81604378630,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":233472,"data_stored":93322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":19,"seq":81604378629,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":233472,"data_stored":93322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":233472,"data_stored":93322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411334,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":233472,"data_stored":93322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411334,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27768,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344072,"statfs":{"total":96636764160,"available":96608329728,"internally_reserved":0,"allocated":765952,"data_stored":622250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411334,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":85534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":18,"seq":77309411332,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":85534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411333,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":85534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":233472,"data_stored":93322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":17,"seq":73014444038,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":119074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":17,"seq":73014444038,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":815104,"data_stored":656359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":17,"seq":73014444038,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":119074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T02:03:14.027 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pgp_num 16 2024-04-19T02:03:14.065 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:14.063+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:14.102 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:14.103 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:14.117 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-19T02:03:14.165 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:14.167+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:14.266 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:14.267+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:14.271 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:14.271 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:14.284 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-19T02:03:14.367 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:14.367+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:14.394 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:14.394 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:14.406 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-19T02:03:14.468 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:14.471+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:14.569 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:14.571+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:14.570 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:14.570 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:14.583 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-19T02:03:14.611 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T02:03:14.618 INFO:teuthology.orchestra.run.smithi059.stderr:set pool 3 pgp_num to 16 2024-04-19T02:03:14.621 INFO:tasks.rados.rados.0.smithi059.stdout:64: done (1 left) 2024-04-19T02:03:14.621 INFO:tasks.rados.rados.0.smithi059.stdout:66: done (0 left) 2024-04-19T02:03:14.621 INFO:tasks.rados.rados.0.smithi059.stdout:67: read oid 14 snap -1 2024-04-19T02:03:14.622 INFO:tasks.rados.rados.0.smithi059.stdout:67: expect (ObjNum 13 snap 0 seq_num 13) 2024-04-19T02:03:14.622 INFO:tasks.rados.rados.0.smithi059.stdout:68: rollback oid 20 current snap is 1 2024-04-19T02:03:14.622 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 20 to 0 2024-04-19T02:03:14.622 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 13 current snap is 1 2024-04-19T02:03:14.622 INFO:tasks.rados.rados.0.smithi059.stdout:69: seq_num 56 ranges {212992=573440,786432=376832} 2024-04-19T02:03:14.625 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is unset 2024-04-19T02:03:14.628 INFO:tasks.rados.rados.0.smithi059.stdout:69: writing smithi05930959-13 from 212992 to 786432 tid 1 2024-04-19T02:03:14.633 INFO:tasks.rados.rados.0.smithi059.stdout:69: writing smithi05930959-13 from 786432 to 1163264 tid 2 2024-04-19T02:03:14.633 INFO:tasks.rados.rados.0.smithi059.stdout:70: rollback oid 48 current snap is 1 2024-04-19T02:03:14.633 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 48 to 0 2024-04-19T02:03:14.633 INFO:tasks.rados.rados.0.smithi059.stdout:67: expect (ObjNum 13 snap 0 seq_num 13) 2024-04-19T02:03:14.647 INFO:tasks.rados.rados.0.smithi059.stdout:68: finishing rollback tid 0 to smithi05930959-20 2024-04-19T02:03:14.647 INFO:tasks.rados.rados.0.smithi059.stdout:68: finishing rollback tid 1 to smithi05930959-20 2024-04-19T02:03:14.648 INFO:tasks.rados.rados.0.smithi059.stdout:68: finishing rollback tid 2 to smithi05930959-20 2024-04-19T02:03:14.648 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 20 v 14 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-19T02:03:14.648 INFO:tasks.rados.rados.0.smithi059.stdout:67: done (3 left) 2024-04-19T02:03:14.648 INFO:tasks.rados.rados.0.smithi059.stdout:68: done (2 left) 2024-04-19T02:03:14.648 INFO:tasks.rados.rados.0.smithi059.stdout:71: copy_from oid 14 from oid 16 current snap is 1 2024-04-19T02:03:14.648 INFO:tasks.rados.rados.0.smithi059.stdout:72: rollback oid 30 current snap is 1 2024-04-19T02:03:14.648 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 30 to 0 2024-04-19T02:03:14.649 INFO:tasks.rados.rados.0.smithi059.stdout:73: rollback oid 37 current snap is 1 2024-04-19T02:03:14.649 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 37 to 0 2024-04-19T02:03:14.649 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 8 current snap is 1 2024-04-19T02:03:14.649 INFO:tasks.rados.rados.0.smithi059.stdout:74: seq_num 57 ranges {131072=589824,720896=376832} 2024-04-19T02:03:14.654 INFO:tasks.rados.rados.0.smithi059.stdout:74: writing smithi05930959-8 from 131072 to 720896 tid 1 2024-04-19T02:03:14.658 INFO:tasks.rados.rados.0.smithi059.stdout:74: writing smithi05930959-8 from 720896 to 1097728 tid 2 2024-04-19T02:03:14.658 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 10 current snap is 1 2024-04-19T02:03:14.658 INFO:tasks.rados.rados.0.smithi059.stdout:75: seq_num 58 ranges {163840=606208,770048=376832} 2024-04-19T02:03:14.662 INFO:tasks.rados.rados.0.smithi059.stdout:75: writing smithi05930959-10 from 163840 to 770048 tid 1 2024-04-19T02:03:14.665 INFO:tasks.rados.rados.0.smithi059.stdout:75: writing smithi05930959-10 from 770048 to 1146880 tid 2 2024-04-19T02:03:14.665 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 1 current snap is 1 2024-04-19T02:03:14.666 INFO:tasks.rados.rados.0.smithi059.stdout:70: finishing rollback tid 0 to smithi05930959-48 2024-04-19T02:03:14.666 INFO:tasks.rados.rados.0.smithi059.stdout:70: finishing rollback tid 1 to smithi05930959-48 2024-04-19T02:03:14.666 INFO:tasks.rados.rados.0.smithi059.stdout:69: finishing write tid 1 to smithi05930959-13 2024-04-19T02:03:14.666 INFO:tasks.rados.rados.0.smithi059.stdout:70: finishing rollback tid 2 to smithi05930959-48 2024-04-19T02:03:14.666 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 48 v 18 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-19T02:03:14.666 INFO:tasks.rados.rados.0.smithi059.stdout:69: finishing write tid 2 to smithi05930959-13 2024-04-19T02:03:14.666 INFO:tasks.rados.rados.0.smithi059.stdout:69: finishing write tid 3 to smithi05930959-13 2024-04-19T02:03:14.666 INFO:tasks.rados.rados.0.smithi059.stdout:73: finishing rollback tid 0 to smithi05930959-37 2024-04-19T02:03:14.666 INFO:tasks.rados.rados.0.smithi059.stdout:72: finishing rollback tid 0 to smithi05930959-30 2024-04-19T02:03:14.666 INFO:tasks.rados.rados.0.smithi059.stdout:73: finishing rollback tid 1 to smithi05930959-37 2024-04-19T02:03:14.666 INFO:tasks.rados.rados.0.smithi059.stdout:72: finishing rollback tid 1 to smithi05930959-30 2024-04-19T02:03:14.666 INFO:tasks.rados.rados.0.smithi059.stdout:71: finishing copy_from racing read to smithi05930959-14 2024-04-19T02:03:14.666 INFO:tasks.rados.rados.0.smithi059.stdout:76: seq_num 59 ranges {32768=622592,655360=376832} 2024-04-19T02:03:14.670 INFO:tasks.rados.rados.0.smithi059.stdout:76: writing smithi05930959-1 from 32768 to 655360 tid 1 2024-04-19T02:03:14.671 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:14.671+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:14.673 INFO:tasks.rados.rados.0.smithi059.stdout:76: writing smithi05930959-1 from 655360 to 1032192 tid 2 2024-04-19T02:03:14.673 INFO:tasks.rados.rados.0.smithi059.stdout:70: done (7 left) 2024-04-19T02:03:14.673 INFO:tasks.rados.rados.0.smithi059.stdout:77: read oid 49 snap -1 2024-04-19T02:03:14.673 INFO:tasks.rados.rados.0.smithi059.stdout:77: expect (ObjNum 48 snap 0 seq_num 48) 2024-04-19T02:03:14.674 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 40 current snap is 1 2024-04-19T02:03:14.674 INFO:tasks.rados.rados.0.smithi059.stdout:73: finishing rollback tid 2 to smithi05930959-37 2024-04-19T02:03:14.674 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 37 v 10 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-19T02:03:14.674 INFO:tasks.rados.rados.0.smithi059.stdout:78: seq_num 60 ranges {671744=638976,1310720=376832} 2024-04-19T02:03:14.682 INFO:tasks.rados.rados.0.smithi059.stdout:78: writing smithi05930959-40 from 671744 to 1310720 tid 1 2024-04-19T02:03:14.685 INFO:tasks.rados.rados.0.smithi059.stdout:78: writing smithi05930959-40 from 1310720 to 1687552 tid 2 2024-04-19T02:03:14.685 INFO:tasks.rados.rados.0.smithi059.stdout:73: done (8 left) 2024-04-19T02:03:14.685 INFO:tasks.rados.rados.0.smithi059.stdout:79: read oid 9 snap -1 2024-04-19T02:03:14.685 INFO:tasks.rados.rados.0.smithi059.stdout:79: expect (ObjNum 8 snap 0 seq_num 8) 2024-04-19T02:03:14.685 INFO:tasks.rados.rados.0.smithi059.stdout:80: read oid 5 snap -1 2024-04-19T02:03:14.686 INFO:tasks.rados.rados.0.smithi059.stdout:80: expect (ObjNum 4 snap 0 seq_num 4) 2024-04-19T02:03:14.686 INFO:tasks.rados.rados.0.smithi059.stdout:81: snap_create 2024-04-19T02:03:14.686 INFO:tasks.rados.rados.0.smithi059.stdout:72: finishing rollback tid 2 to smithi05930959-30 2024-04-19T02:03:14.686 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 30 v 11 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-19T02:03:14.686 INFO:tasks.rados.rados.0.smithi059.stdout:71: finishing copy_from to smithi05930959-14 2024-04-19T02:03:14.686 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 14 v 4 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-19T02:03:14.686 INFO:tasks.rados.rados.0.smithi059.stdout:74: finishing write tid 1 to smithi05930959-8 2024-04-19T02:03:14.686 INFO:tasks.rados.rados.0.smithi059.stdout:75: finishing write tid 1 to smithi05930959-10 2024-04-19T02:03:14.686 INFO:tasks.rados.rados.0.smithi059.stdout:69: finishing write tid 4 to smithi05930959-13 2024-04-19T02:03:14.686 INFO:tasks.rados.rados.0.smithi059.stdout:69: oid 13 updating version 0 to 11 2024-04-19T02:03:14.686 INFO:tasks.rados.rados.0.smithi059.stdout:69: oid 13 version 11 is already newer than 9 2024-04-19T02:03:14.686 INFO:tasks.rados.rados.0.smithi059.stdout:69: oid 13 version 11 is already newer than 10 2024-04-19T02:03:14.686 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 13 v 11 (ObjNum 56 snap 1 seq_num 56) dirty exists 2024-04-19T02:03:14.686 INFO:tasks.rados.rados.0.smithi059.stdout:69: left oid 13 (ObjNum 56 snap 1 seq_num 56) 2024-04-19T02:03:14.687 INFO:tasks.rados.rados.0.smithi059.stdout:75: finishing write tid 2 to smithi05930959-10 2024-04-19T02:03:14.687 INFO:tasks.rados.rados.0.smithi059.stdout:75: finishing write tid 3 to smithi05930959-10 2024-04-19T02:03:14.687 INFO:tasks.rados.rados.0.smithi059.stdout:74: finishing write tid 2 to smithi05930959-8 2024-04-19T02:03:14.687 INFO:tasks.rados.rados.0.smithi059.stdout:74: finishing write tid 3 to smithi05930959-8 2024-04-19T02:03:14.687 INFO:tasks.rados.rados.0.smithi059.stdout:75: finishing write tid 4 to smithi05930959-10 2024-04-19T02:03:14.687 INFO:tasks.rados.rados.0.smithi059.stdout:75: oid 10 updating version 0 to 21 2024-04-19T02:03:14.687 INFO:tasks.rados.rados.0.smithi059.stdout:75: oid 10 updating version 21 to 22 2024-04-19T02:03:14.687 INFO:tasks.rados.rados.0.smithi059.stdout:75: oid 10 version 22 is already newer than 20 2024-04-19T02:03:14.687 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 10 v 22 (ObjNum 58 snap 1 seq_num 58) dirty exists 2024-04-19T02:03:14.687 INFO:tasks.rados.rados.0.smithi059.stdout:75: left oid 10 (ObjNum 58 snap 1 seq_num 58) 2024-04-19T02:03:14.687 INFO:tasks.rados.rados.0.smithi059.stdout:76: finishing write tid 1 to smithi05930959-1 2024-04-19T02:03:14.690 INFO:tasks.rados.rados.0.smithi059.stdout:76: finishing write tid 2 to smithi05930959-1 2024-04-19T02:03:14.691 INFO:tasks.rados.rados.0.smithi059.stdout:76: finishing write tid 3 to smithi05930959-1 2024-04-19T02:03:14.696 INFO:tasks.rados.rados.0.smithi059.stdout:76: finishing write tid 4 to smithi05930959-1 2024-04-19T02:03:14.696 INFO:tasks.rados.rados.0.smithi059.stdout:76: oid 1 updating version 0 to 11 2024-04-19T02:03:14.696 INFO:tasks.rados.rados.0.smithi059.stdout:76: oid 1 version 11 is already newer than 9 2024-04-19T02:03:14.696 INFO:tasks.rados.rados.0.smithi059.stdout:76: oid 1 version 11 is already newer than 10 2024-04-19T02:03:14.696 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 1 v 11 (ObjNum 59 snap 1 seq_num 59) dirty exists 2024-04-19T02:03:14.697 INFO:tasks.rados.rados.0.smithi059.stdout:76: left oid 1 (ObjNum 59 snap 1 seq_num 59) 2024-04-19T02:03:14.697 INFO:tasks.rados.rados.0.smithi059.stdout:80: expect (ObjNum 4 snap 0 seq_num 4) 2024-04-19T02:03:14.700 INFO:tasks.rados.rados.0.smithi059.stdout:74: finishing write tid 4 to smithi05930959-8 2024-04-19T02:03:14.700 INFO:tasks.rados.rados.0.smithi059.stdout:74: oid 8 updating version 0 to 14 2024-04-19T02:03:14.700 INFO:tasks.rados.rados.0.smithi059.stdout:74: oid 8 version 14 is already newer than 13 2024-04-19T02:03:14.700 INFO:tasks.rados.rados.0.smithi059.stdout:74: oid 8 updating version 14 to 15 2024-04-19T02:03:14.700 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 8 v 15 (ObjNum 57 snap 1 seq_num 57) dirty exists 2024-04-19T02:03:14.700 INFO:tasks.rados.rados.0.smithi059.stdout:74: left oid 8 (ObjNum 57 snap 1 seq_num 57) 2024-04-19T02:03:14.700 INFO:tasks.rados.rados.0.smithi059.stdout:77: expect (ObjNum 48 snap 0 seq_num 48) 2024-04-19T02:03:14.737 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:14.738 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:14.739 INFO:tasks.rados.rados.0.smithi059.stdout:78: finishing write tid 1 to smithi05930959-40 2024-04-19T02:03:14.739 INFO:tasks.rados.rados.0.smithi059.stdout:78: finishing write tid 2 to smithi05930959-40 2024-04-19T02:03:14.739 INFO:tasks.rados.rados.0.smithi059.stdout:78: finishing write tid 3 to smithi05930959-40 2024-04-19T02:03:14.739 INFO:tasks.rados.rados.0.smithi059.stdout:79: expect (ObjNum 8 snap 0 seq_num 8) 2024-04-19T02:03:14.746 INFO:tasks.rados.rados.0.smithi059.stdout:78: finishing write tid 4 to smithi05930959-40 2024-04-19T02:03:14.746 INFO:tasks.rados.rados.0.smithi059.stdout:78: oid 40 updating version 0 to 10 2024-04-19T02:03:14.746 INFO:tasks.rados.rados.0.smithi059.stdout:78: oid 40 updating version 10 to 11 2024-04-19T02:03:14.746 INFO:tasks.rados.rados.0.smithi059.stdout:78: oid 40 updating version 11 to 12 2024-04-19T02:03:14.746 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 40 v 12 (ObjNum 60 snap 1 seq_num 60) dirty exists 2024-04-19T02:03:14.746 INFO:tasks.rados.rados.0.smithi059.stdout:78: left oid 40 (ObjNum 60 snap 1 seq_num 60) 2024-04-19T02:03:14.753 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-19T02:03:14.771 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:14.771+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:14.776 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-04-19T02:03:14.776 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T02:03:14.794 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-04-19T02:03:14.794 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T02:03:14.810 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-04-19T02:03:14.810 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T02:03:14.826 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-04-19T02:03:14.827 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T02:03:14.847 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-04-19T02:03:14.847 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T02:03:14.867 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-04-19T02:03:14.868 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T02:03:14.872 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:14.871+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:14.888 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:14.888 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-04-19T02:03:14.888 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-04-19T02:03:14.888 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [] 2024-04-19T02:03:14.888 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:14.888 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-04-19T02:03:14.889 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T02:03:14.901 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-19T02:03:14.908 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-04-19T02:03:14.909 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T02:03:14.931 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-04-19T02:03:14.931 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T02:03:14.952 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-04-19T02:03:14.953 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T02:03:14.974 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:14.975+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:14.974 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-04-19T02:03:14.974 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T02:03:14.997 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-04-19T02:03:14.997 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T02:03:15.032 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:15.032 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:15.032 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 0 2024-04-19T02:03:15.032 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:15.046 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-19T02:03:15.089 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:15.091+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:15.164 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:15.164 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:15.164 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-04-19T02:03:15.164 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-04-19T02:03:15.164 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:15.175 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:15.175+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:15.178 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-19T02:03:15.276 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:15.279+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:15.346 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:15.346 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-04-19T02:03:15.346 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-04-19T02:03:15.346 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [] 2024-04-19T02:03:15.346 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:15.360 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-19T02:03:15.377 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:15.379+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:15.476 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:15.477 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:15.477 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 0 2024-04-19T02:03:15.477 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:15.478 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:15.479+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:15.490 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-19T02:03:15.579 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:15.579+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:15.613 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:15.614 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:15.614 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-04-19T02:03:15.614 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-04-19T02:03:15.614 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:15.629 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-19T02:03:15.629 INFO:tasks.rados.rados.0.smithi059.stdout:69: done (10 left) 2024-04-19T02:03:15.630 INFO:tasks.rados.rados.0.smithi059.stdout:71: done (9 left) 2024-04-19T02:03:15.630 INFO:tasks.rados.rados.0.smithi059.stdout:72: done (8 left) 2024-04-19T02:03:15.630 INFO:tasks.rados.rados.0.smithi059.stdout:74: done (7 left) 2024-04-19T02:03:15.630 INFO:tasks.rados.rados.0.smithi059.stdout:75: done (6 left) 2024-04-19T02:03:15.630 INFO:tasks.rados.rados.0.smithi059.stdout:76: done (5 left) 2024-04-19T02:03:15.630 INFO:tasks.rados.rados.0.smithi059.stdout:77: done (4 left) 2024-04-19T02:03:15.630 INFO:tasks.rados.rados.0.smithi059.stdout:78: done (3 left) 2024-04-19T02:03:15.630 INFO:tasks.rados.rados.0.smithi059.stdout:79: done (2 left) 2024-04-19T02:03:15.630 INFO:tasks.rados.rados.0.smithi059.stdout:80: done (1 left) 2024-04-19T02:03:15.630 INFO:tasks.rados.rados.0.smithi059.stdout:81: done (0 left) 2024-04-19T02:03:15.630 INFO:tasks.rados.rados.0.smithi059.stdout:82: delete oid 27 current snap is 2 2024-04-19T02:03:15.644 INFO:tasks.rados.rados.0.smithi059.stdout:82: done (0 left) 2024-04-19T02:03:15.644 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 23 current snap is 2 2024-04-19T02:03:15.644 INFO:tasks.rados.rados.0.smithi059.stdout:83: seq_num 62 ranges {376832=671744,1048576=376832} 2024-04-19T02:03:15.652 INFO:tasks.rados.rados.0.smithi059.stdout:83: writing smithi05930959-23 from 376832 to 1048576 tid 1 2024-04-19T02:03:15.655 INFO:tasks.rados.rados.0.smithi059.stdout:83: writing smithi05930959-23 from 1048576 to 1425408 tid 2 2024-04-19T02:03:15.655 INFO:tasks.rados.rados.0.smithi059.stdout:84: snap_remove snap 1 2024-04-19T02:03:15.721 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:15.723+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:15.781 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:15.779+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:15.798 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:15.798 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-04-19T02:03:15.798 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-04-19T02:03:15.798 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-04-19T02:03:15.798 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.798 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:21 1.0 e19/18 1:30306672:devicehealth::main.db.0000000000000000:head v 19'11, mlcod=19'8)", 2024-04-19T02:03:15.798 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:00.012811+0000", 2024-04-19T02:03:15.799 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 15.78779276, 2024-04-19T02:03:15.799 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0053931220000000002, 2024-04-19T02:03:15.799 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:15.799 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:15.799 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:15.799 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:15.799 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.799 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:15.799 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.012811+0000", 2024-04-19T02:03:15.799 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.799 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.799 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.799 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:15.799 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.012811+0000", 2024-04-19T02:03:15.800 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.800 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.800 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.800 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:15.800 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.012812+0000", 2024-04-19T02:03:15.800 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7899999999999998e-07 2024-04-19T02:03:15.800 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.800 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.800 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:15.800 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.012925+0000", 2024-04-19T02:03:15.800 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011290699999999999 2024-04-19T02:03:15.800 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.800 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.800 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:15.800 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.012934+0000", 2024-04-19T02:03:15.800 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.611e-06 2024-04-19T02:03:15.801 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.801 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.801 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:15.801 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.012940+0000", 2024-04-19T02:03:15.801 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4399999999999996e-06 2024-04-19T02:03:15.801 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.801 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.801 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:15.801 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.013020+0000", 2024-04-19T02:03:15.801 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9740999999999994e-05 2024-04-19T02:03:15.801 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.801 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.801 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:15.801 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.013056+0000", 2024-04-19T02:03:15.801 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6247e-05 2024-04-19T02:03:15.802 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.802 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.802 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:15.802 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.018113+0000", 2024-04-19T02:03:15.802 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0050570179999999999 2024-04-19T02:03:15.802 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.802 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.802 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:15.802 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.018204+0000", 2024-04-19T02:03:15.802 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1478999999999996e-05 2024-04-19T02:03:15.802 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.802 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:15.802 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.802 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.803 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.803 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:22 1.0 e19/18 1:30306672:devicehealth::main.db.0000000000000000:head v 19'12, mlcod=19'8)", 2024-04-19T02:03:15.803 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:00.014091+0000", 2024-04-19T02:03:15.803 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 15.786512639, 2024-04-19T02:03:15.803 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0047229300000000002, 2024-04-19T02:03:15.803 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:15.803 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:15.803 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:15.803 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:15.803 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.803 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:15.803 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.014091+0000", 2024-04-19T02:03:15.803 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.803 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.803 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.803 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:15.804 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.014091+0000", 2024-04-19T02:03:15.804 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.804 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.804 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.804 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:15.804 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.014092+0000", 2024-04-19T02:03:15.804 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.75e-07 2024-04-19T02:03:15.804 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.804 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.804 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:15.804 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.014097+0000", 2024-04-19T02:03:15.804 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7709999999999997e-06 2024-04-19T02:03:15.804 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.804 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.804 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:15.804 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.014106+0000", 2024-04-19T02:03:15.804 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7210000000000006e-06 2024-04-19T02:03:15.805 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.805 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.805 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:15.805 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.014112+0000", 2024-04-19T02:03:15.805 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.22e-06 2024-04-19T02:03:15.805 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.805 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.805 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:15.805 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.014174+0000", 2024-04-19T02:03:15.805 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2843999999999997e-05 2024-04-19T02:03:15.805 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.805 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.805 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:15.805 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.014215+0000", 2024-04-19T02:03:15.805 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0083999999999998e-05 2024-04-19T02:03:15.805 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.806 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.806 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:15.806 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.018755+0000", 2024-04-19T02:03:15.806 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0045405619999999997 2024-04-19T02:03:15.806 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.806 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.806 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:15.806 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.018814+0000", 2024-04-19T02:03:15.806 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9052999999999999e-05 2024-04-19T02:03:15.806 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.806 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:15.806 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.806 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.806 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.807 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:84 1.0 e19/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'54, mlcod=19'50)", 2024-04-19T02:03:15.807 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:00.136060+0000", 2024-04-19T02:03:15.807 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 15.664543922, 2024-04-19T02:03:15.807 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0051945680000000001, 2024-04-19T02:03:15.807 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:15.807 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:15.807 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:15.807 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:15.807 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.807 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:15.807 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.136060+0000", 2024-04-19T02:03:15.807 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.807 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.807 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.807 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:15.808 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.136060+0000", 2024-04-19T02:03:15.808 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.808 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.808 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.808 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:15.808 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.136062+0000", 2024-04-19T02:03:15.808 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7600000000000001e-06 2024-04-19T02:03:15.808 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.808 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.808 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:15.808 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.136120+0000", 2024-04-19T02:03:15.808 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7834000000000002e-05 2024-04-19T02:03:15.808 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.808 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.808 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:15.808 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.136148+0000", 2024-04-19T02:03:15.809 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8062999999999999e-05 2024-04-19T02:03:15.809 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.809 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.809 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:15.809 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.136161+0000", 2024-04-19T02:03:15.809 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3033e-05 2024-04-19T02:03:15.809 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.809 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.809 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:15.809 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.136252+0000", 2024-04-19T02:03:15.809 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1552999999999996e-05 2024-04-19T02:03:15.809 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.809 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.809 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:15.809 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.136284+0000", 2024-04-19T02:03:15.809 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1893000000000001e-05 2024-04-19T02:03:15.810 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.810 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.810 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:15.810 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.141186+0000", 2024-04-19T02:03:15.810 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0049016119999999996 2024-04-19T02:03:15.810 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.810 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.810 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:15.810 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.141255+0000", 2024-04-19T02:03:15.810 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8819999999999995e-05 2024-04-19T02:03:15.810 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.810 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:15.810 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.810 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.810 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.811 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:85 1.0 e19/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'55, mlcod=19'51)", 2024-04-19T02:03:15.811 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:00.137394+0000", 2024-04-19T02:03:15.811 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 15.663210288, 2024-04-19T02:03:15.811 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0064003139999999998, 2024-04-19T02:03:15.811 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:15.811 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:15.811 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:15.811 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:15.811 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.811 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:15.811 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.137394+0000", 2024-04-19T02:03:15.811 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.811 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.811 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.811 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:15.812 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.137394+0000", 2024-04-19T02:03:15.812 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.812 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.812 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.812 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:15.812 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.137395+0000", 2024-04-19T02:03:15.812 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.156e-06 2024-04-19T02:03:15.812 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.812 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.812 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:15.812 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.137403+0000", 2024-04-19T02:03:15.812 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2740000000000007e-06 2024-04-19T02:03:15.812 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.812 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.812 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:15.813 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.137421+0000", 2024-04-19T02:03:15.813 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7782999999999999e-05 2024-04-19T02:03:15.813 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.813 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.813 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:15.813 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.137481+0000", 2024-04-19T02:03:15.813 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0652000000000001e-05 2024-04-19T02:03:15.813 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.813 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.813 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:15.813 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.137598+0000", 2024-04-19T02:03:15.813 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011613299999999999 2024-04-19T02:03:15.813 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.813 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.813 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:15.813 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.137654+0000", 2024-04-19T02:03:15.814 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6390000000000001e-05 2024-04-19T02:03:15.814 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.814 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.814 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:15.814 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.143681+0000", 2024-04-19T02:03:15.814 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0060272140000000004 2024-04-19T02:03:15.814 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.814 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.814 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:15.814 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.143794+0000", 2024-04-19T02:03:15.814 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000112712 2024-04-19T02:03:15.814 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.814 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:15.814 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.814 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.814 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.815 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:86 1.0 e19/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'56, mlcod=19'51)", 2024-04-19T02:03:15.815 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:00.138559+0000", 2024-04-19T02:03:15.815 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 15.662044931000001, 2024-04-19T02:03:15.815 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0053320930000000004, 2024-04-19T02:03:15.815 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:15.815 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:15.815 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:15.815 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:15.815 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.815 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:15.815 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.138559+0000", 2024-04-19T02:03:15.815 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.815 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.815 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.815 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:15.816 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.138559+0000", 2024-04-19T02:03:15.816 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.816 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.816 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.816 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:15.816 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.138560+0000", 2024-04-19T02:03:15.816 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.44e-06 2024-04-19T02:03:15.816 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.816 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.816 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:15.816 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.138570+0000", 2024-04-19T02:03:15.816 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6139999999999998e-06 2024-04-19T02:03:15.816 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.816 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.816 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:15.816 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.138598+0000", 2024-04-19T02:03:15.817 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8146000000000001e-05 2024-04-19T02:03:15.817 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.817 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.817 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:15.817 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.138611+0000", 2024-04-19T02:03:15.817 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2807e-05 2024-04-19T02:03:15.817 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.817 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.817 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:15.817 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.138698+0000", 2024-04-19T02:03:15.817 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.6872000000000004e-05 2024-04-19T02:03:15.817 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.817 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.817 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:15.817 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.138727+0000", 2024-04-19T02:03:15.817 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8739999999999999e-05 2024-04-19T02:03:15.818 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.818 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.818 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:15.818 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.143800+0000", 2024-04-19T02:03:15.818 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0050737680000000002 2024-04-19T02:03:15.818 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.818 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.818 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:15.818 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.143891+0000", 2024-04-19T02:03:15.818 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0705999999999995e-05 2024-04-19T02:03:15.818 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.818 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:15.818 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.818 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.818 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.819 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:87 1.0 e19/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'57, mlcod=19'51)", 2024-04-19T02:03:15.819 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:00.139853+0000", 2024-04-19T02:03:15.819 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 15.660750474, 2024-04-19T02:03:15.819 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0087466390000000005, 2024-04-19T02:03:15.819 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:15.819 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:15.819 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:15.819 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:15.819 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.819 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:15.819 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.139853+0000", 2024-04-19T02:03:15.819 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.819 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.819 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.819 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:15.820 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.139853+0000", 2024-04-19T02:03:15.820 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.820 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.820 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.820 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:15.820 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.139855+0000", 2024-04-19T02:03:15.820 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.733e-06 2024-04-19T02:03:15.820 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.820 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.820 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:15.820 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.139912+0000", 2024-04-19T02:03:15.820 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6849999999999999e-05 2024-04-19T02:03:15.820 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.820 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.820 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:15.820 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.139940+0000", 2024-04-19T02:03:15.820 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7883000000000001e-05 2024-04-19T02:03:15.821 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.821 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.821 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:15.821 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.139953+0000", 2024-04-19T02:03:15.821 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.314e-05 2024-04-19T02:03:15.821 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.821 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.821 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:15.821 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.140052+0000", 2024-04-19T02:03:15.821 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9246000000000005e-05 2024-04-19T02:03:15.821 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.821 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.821 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:15.821 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.140084+0000", 2024-04-19T02:03:15.821 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1409999999999999e-05 2024-04-19T02:03:15.821 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.822 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.822 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:15.822 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.148540+0000", 2024-04-19T02:03:15.822 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0084567979999999997 2024-04-19T02:03:15.822 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.822 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.822 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:15.822 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.148600+0000", 2024-04-19T02:03:15.822 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9579e-05 2024-04-19T02:03:15.822 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.822 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:15.822 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.822 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.822 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.822 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:88 1.0 e19/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'58, mlcod=19'52)", 2024-04-19T02:03:15.822 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:00.141581+0000", 2024-04-19T02:03:15.823 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 15.659022647, 2024-04-19T02:03:15.823 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0070620880000000002, 2024-04-19T02:03:15.823 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:15.823 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:15.823 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:15.823 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:15.823 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.823 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:15.823 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.141581+0000", 2024-04-19T02:03:15.823 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.823 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.823 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.823 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:15.823 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.141581+0000", 2024-04-19T02:03:15.823 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.823 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.824 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.824 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:15.824 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.141582+0000", 2024-04-19T02:03:15.824 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.147e-06 2024-04-19T02:03:15.824 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.824 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.824 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:15.824 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.141590+0000", 2024-04-19T02:03:15.824 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4229999999999999e-06 2024-04-19T02:03:15.824 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.824 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.824 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:15.824 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.141609+0000", 2024-04-19T02:03:15.824 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9357000000000001e-05 2024-04-19T02:03:15.824 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.824 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.825 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:15.825 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.141642+0000", 2024-04-19T02:03:15.825 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.2342999999999998e-05 2024-04-19T02:03:15.825 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.825 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.825 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:15.825 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.141734+0000", 2024-04-19T02:03:15.825 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.2091999999999999e-05 2024-04-19T02:03:15.825 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.825 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.825 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:15.825 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.141771+0000", 2024-04-19T02:03:15.825 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6913999999999999e-05 2024-04-19T02:03:15.825 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.825 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.825 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:15.826 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.148606+0000", 2024-04-19T02:03:15.826 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0068350030000000001 2024-04-19T02:03:15.826 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.826 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.826 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:15.826 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.148643+0000", 2024-04-19T02:03:15.826 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7808999999999997e-05 2024-04-19T02:03:15.826 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.826 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:15.826 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.826 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.826 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.826 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:89 1.0 e19/18 1:30306672:devicehealth::main.db.0000000000000000:head v 19'59, mlcod=19'57)", 2024-04-19T02:03:15.826 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:00.151382+0000", 2024-04-19T02:03:15.826 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 15.649222355999999, 2024-04-19T02:03:15.826 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0037983600000000002, 2024-04-19T02:03:15.827 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:15.827 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:15.827 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:15.827 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:15.827 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.827 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:15.827 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.151382+0000", 2024-04-19T02:03:15.827 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.827 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.827 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.827 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:15.827 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.151382+0000", 2024-04-19T02:03:15.827 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.827 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.827 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.828 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:15.828 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.151383+0000", 2024-04-19T02:03:15.828 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6530000000000001e-06 2024-04-19T02:03:15.828 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.828 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.828 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:15.828 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.151499+0000", 2024-04-19T02:03:15.828 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000116076 2024-04-19T02:03:15.828 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.828 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.828 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:15.828 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.151527+0000", 2024-04-19T02:03:15.828 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.794e-05 2024-04-19T02:03:15.828 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.828 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.829 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:15.829 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.151540+0000", 2024-04-19T02:03:15.829 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3003000000000001e-05 2024-04-19T02:03:15.829 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.829 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.829 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:15.829 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.151653+0000", 2024-04-19T02:03:15.829 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000112726 2024-04-19T02:03:15.829 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.829 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.829 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:15.829 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.151688+0000", 2024-04-19T02:03:15.829 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5403e-05 2024-04-19T02:03:15.829 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.829 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.829 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:15.830 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.155069+0000", 2024-04-19T02:03:15.830 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0033803570000000001 2024-04-19T02:03:15.830 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.830 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.830 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:15.830 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.155180+0000", 2024-04-19T02:03:15.830 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000111202 2024-04-19T02:03:15.830 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.830 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:15.830 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.830 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.830 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.830 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:106 1.0 e19/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=19'64)", 2024-04-19T02:03:15.830 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:00.314828+0000", 2024-04-19T02:03:15.830 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 15.485775521000001, 2024-04-19T02:03:15.831 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0045193550000000001, 2024-04-19T02:03:15.831 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:15.831 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:15.831 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:15.831 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:15.831 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.831 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:15.831 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.314828+0000", 2024-04-19T02:03:15.831 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.831 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.831 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.831 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:15.831 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.314828+0000", 2024-04-19T02:03:15.831 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.831 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.831 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.832 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:15.832 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.314830+0000", 2024-04-19T02:03:15.832 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.733e-06 2024-04-19T02:03:15.832 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.832 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.832 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:15.832 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.314889+0000", 2024-04-19T02:03:15.832 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8616000000000003e-05 2024-04-19T02:03:15.832 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.832 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.832 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:15.832 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.314917+0000", 2024-04-19T02:03:15.832 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8496999999999998e-05 2024-04-19T02:03:15.832 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.832 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.833 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:15.833 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.314930+0000", 2024-04-19T02:03:15.833 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2607e-05 2024-04-19T02:03:15.833 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.833 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.833 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:15.833 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.315030+0000", 2024-04-19T02:03:15.833 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010015500000000001 2024-04-19T02:03:15.833 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.833 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.833 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:15.833 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.315062+0000", 2024-04-19T02:03:15.833 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1659999999999998e-05 2024-04-19T02:03:15.833 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.833 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.833 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:15.834 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.319250+0000", 2024-04-19T02:03:15.834 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0041880779999999996 2024-04-19T02:03:15.834 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.834 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.834 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:15.834 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.319348+0000", 2024-04-19T02:03:15.834 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8009000000000003e-05 2024-04-19T02:03:15.834 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.834 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:15.834 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.834 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.834 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.834 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:107 1.0 e19/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=19'64)", 2024-04-19T02:03:15.834 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:00.315743+0000", 2024-04-19T02:03:15.835 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 15.484860508000001, 2024-04-19T02:03:15.835 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003676178, 2024-04-19T02:03:15.835 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:15.835 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:15.835 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:15.835 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:15.835 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.835 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:15.835 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.315743+0000", 2024-04-19T02:03:15.835 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.835 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.836 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.836 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:15.836 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.315743+0000", 2024-04-19T02:03:15.836 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.836 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.836 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.836 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:15.836 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.315745+0000", 2024-04-19T02:03:15.836 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.73e-06 2024-04-19T02:03:15.836 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.836 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.836 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:15.836 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.315757+0000", 2024-04-19T02:03:15.836 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2089999999999999e-05 2024-04-19T02:03:15.836 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.837 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.837 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:15.837 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.315786+0000", 2024-04-19T02:03:15.837 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8333000000000001e-05 2024-04-19T02:03:15.837 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.837 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.837 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:15.837 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.315799+0000", 2024-04-19T02:03:15.837 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3013000000000001e-05 2024-04-19T02:03:15.837 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.837 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.837 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:15.837 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.315889+0000", 2024-04-19T02:03:15.837 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0296000000000004e-05 2024-04-19T02:03:15.837 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.838 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.838 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:15.838 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.315916+0000", 2024-04-19T02:03:15.838 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7596000000000001e-05 2024-04-19T02:03:15.838 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.838 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.838 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:15.838 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.319353+0000", 2024-04-19T02:03:15.838 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0034365440000000001 2024-04-19T02:03:15.838 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.838 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.838 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:15.838 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.319420+0000", 2024-04-19T02:03:15.838 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6575999999999994e-05 2024-04-19T02:03:15.839 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.839 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:15.839 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.839 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.839 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.839 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:113 1.0 e19/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'75, mlcod=19'71)", 2024-04-19T02:03:15.839 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:00.323587+0000", 2024-04-19T02:03:15.839 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 15.477017189, 2024-04-19T02:03:15.839 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0058327889999999997, 2024-04-19T02:03:15.839 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:15.842 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:15.842 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:15.842 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:15.842 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.842 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:15.842 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.323587+0000", 2024-04-19T02:03:15.842 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.842 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.842 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.842 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:15.842 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.323587+0000", 2024-04-19T02:03:15.842 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.842 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.842 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.843 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:15.843 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.323589+0000", 2024-04-19T02:03:15.843 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.934e-06 2024-04-19T02:03:15.843 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.843 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.843 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:15.843 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.323601+0000", 2024-04-19T02:03:15.843 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2683e-05 2024-04-19T02:03:15.843 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.843 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.843 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:15.843 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.323629+0000", 2024-04-19T02:03:15.843 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7823000000000001e-05 2024-04-19T02:03:15.843 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.843 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.844 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:15.844 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.323642+0000", 2024-04-19T02:03:15.844 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3169999999999999e-05 2024-04-19T02:03:15.844 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.844 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.844 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:15.844 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.323762+0000", 2024-04-19T02:03:15.844 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011994600000000001 2024-04-19T02:03:15.844 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.844 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.844 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:15.844 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.323814+0000", 2024-04-19T02:03:15.844 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1430000000000001e-05 2024-04-19T02:03:15.845 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.845 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.845 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:15.845 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.329316+0000", 2024-04-19T02:03:15.845 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00550257 2024-04-19T02:03:15.845 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.845 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.845 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:15.845 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.329419+0000", 2024-04-19T02:03:15.845 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010323300000000001 2024-04-19T02:03:15.845 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.845 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:15.845 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.845 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.845 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.846 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:114 1.0 e19/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'76, mlcod=19'72)", 2024-04-19T02:03:15.846 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:00.324817+0000", 2024-04-19T02:03:15.846 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 15.475787144, 2024-04-19T02:03:15.846 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0057275679999999997, 2024-04-19T02:03:15.846 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:15.846 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:15.846 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:15.846 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:15.846 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.846 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:15.846 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.324817+0000", 2024-04-19T02:03:15.846 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.846 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.846 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.846 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:15.847 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.324817+0000", 2024-04-19T02:03:15.847 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.847 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.847 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.847 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:15.847 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.324818+0000", 2024-04-19T02:03:15.847 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6559999999999999e-06 2024-04-19T02:03:15.847 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.847 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.847 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:15.847 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.324831+0000", 2024-04-19T02:03:15.847 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.242e-05 2024-04-19T02:03:15.847 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.847 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.847 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:15.848 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.324855+0000", 2024-04-19T02:03:15.848 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4300000000000001e-05 2024-04-19T02:03:15.848 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.848 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.848 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:15.848 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.324869+0000", 2024-04-19T02:03:15.848 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3713000000000001e-05 2024-04-19T02:03:15.848 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.848 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.848 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:15.848 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.324973+0000", 2024-04-19T02:03:15.848 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000104183 2024-04-19T02:03:15.848 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.848 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.848 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:15.848 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.325009+0000", 2024-04-19T02:03:15.849 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5698999999999998e-05 2024-04-19T02:03:15.849 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.849 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.849 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:15.849 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.330447+0000", 2024-04-19T02:03:15.849 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0054381949999999998 2024-04-19T02:03:15.849 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.849 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.849 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:15.849 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.330544+0000", 2024-04-19T02:03:15.849 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7402e-05 2024-04-19T02:03:15.849 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.849 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:15.849 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.850 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.850 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.850 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:115 1.0 e19/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'77, mlcod=19'72)", 2024-04-19T02:03:15.850 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:00.325999+0000", 2024-04-19T02:03:15.850 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 15.474604674, 2024-04-19T02:03:15.850 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0046238750000000004, 2024-04-19T02:03:15.850 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:15.850 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:15.850 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:15.850 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:15.850 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.850 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:15.851 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.325999+0000", 2024-04-19T02:03:15.851 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.851 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.851 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.851 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:15.851 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.325999+0000", 2024-04-19T02:03:15.851 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.851 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.851 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.851 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:15.851 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.326001+0000", 2024-04-19T02:03:15.851 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.73e-06 2024-04-19T02:03:15.851 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.852 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.852 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:15.852 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.326057+0000", 2024-04-19T02:03:15.852 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5723000000000002e-05 2024-04-19T02:03:15.852 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.852 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.852 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:15.852 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.326085+0000", 2024-04-19T02:03:15.852 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8197000000000001e-05 2024-04-19T02:03:15.852 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.852 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.852 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:15.853 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.326098+0000", 2024-04-19T02:03:15.853 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2819999999999999e-05 2024-04-19T02:03:15.853 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.853 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.853 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:15.853 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.328527+0000", 2024-04-19T02:03:15.853 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0024289139999999999 2024-04-19T02:03:15.853 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.853 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.853 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:15.853 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.328557+0000", 2024-04-19T02:03:15.853 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0586000000000002e-05 2024-04-19T02:03:15.853 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.854 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.854 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:15.854 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.330552+0000", 2024-04-19T02:03:15.854 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0019943679999999998 2024-04-19T02:03:15.854 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.854 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.854 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:15.854 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.330623+0000", 2024-04-19T02:03:15.854 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1537000000000003e-05 2024-04-19T02:03:15.854 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.854 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:15.855 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.855 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.855 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.855 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:117 1.0 e19/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'79, mlcod=19'73)", 2024-04-19T02:03:15.855 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:00.328486+0000", 2024-04-19T02:03:15.855 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 15.472117853, 2024-04-19T02:03:15.855 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0051412910000000001, 2024-04-19T02:03:15.855 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:15.855 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:15.855 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:15.855 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:15.855 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.856 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:15.856 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.328486+0000", 2024-04-19T02:03:15.856 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.856 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.856 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.856 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:15.856 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.328486+0000", 2024-04-19T02:03:15.856 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.856 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.856 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.856 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:15.856 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.328487+0000", 2024-04-19T02:03:15.857 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0359999999999999e-06 2024-04-19T02:03:15.857 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.857 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.857 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:15.857 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.328494+0000", 2024-04-19T02:03:15.857 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4139999999999997e-06 2024-04-19T02:03:15.857 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.857 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.857 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:15.857 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.328508+0000", 2024-04-19T02:03:15.857 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.308e-05 2024-04-19T02:03:15.857 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.857 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.858 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:15.858 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.328516+0000", 2024-04-19T02:03:15.858 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7960000000000001e-06 2024-04-19T02:03:15.858 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.858 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.858 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:15.858 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.329976+0000", 2024-04-19T02:03:15.858 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0014596089999999999 2024-04-19T02:03:15.858 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.858 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.858 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:15.859 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.330029+0000", 2024-04-19T02:03:15.859 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2899000000000003e-05 2024-04-19T02:03:15.859 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.859 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.859 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:15.859 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.333536+0000", 2024-04-19T02:03:15.859 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0035075380000000001 2024-04-19T02:03:15.859 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.859 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.859 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:15.859 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.333627+0000", 2024-04-19T02:03:15.859 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0919000000000001e-05 2024-04-19T02:03:15.859 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.860 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:15.860 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.860 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.860 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.860 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:119 1.0 e19/18 1:30306672:devicehealth::main.db.0000000000000000:head v 19'81, mlcod=19'78)", 2024-04-19T02:03:15.860 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:00.336987+0000", 2024-04-19T02:03:15.860 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 15.463616571999999, 2024-04-19T02:03:15.860 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0036515050000000002, 2024-04-19T02:03:15.860 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:15.860 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:15.860 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:15.860 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:15.860 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.861 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:15.861 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.336987+0000", 2024-04-19T02:03:15.861 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.861 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.861 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.861 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:15.861 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.336987+0000", 2024-04-19T02:03:15.861 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.861 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.861 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.861 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:15.861 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.336989+0000", 2024-04-19T02:03:15.862 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7540000000000001e-06 2024-04-19T02:03:15.862 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.862 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.862 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:15.862 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.337107+0000", 2024-04-19T02:03:15.862 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000117559 2024-04-19T02:03:15.862 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.862 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.862 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:15.862 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.337131+0000", 2024-04-19T02:03:15.862 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4043e-05 2024-04-19T02:03:15.863 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.863 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.863 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:15.863 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.337143+0000", 2024-04-19T02:03:15.863 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2737e-05 2024-04-19T02:03:15.863 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.863 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.863 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:15.863 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.337266+0000", 2024-04-19T02:03:15.863 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012231500000000001 2024-04-19T02:03:15.863 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.863 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.864 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:15.864 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.337325+0000", 2024-04-19T02:03:15.864 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9406e-05 2024-04-19T02:03:15.864 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.864 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.864 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:15.864 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.340539+0000", 2024-04-19T02:03:15.864 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003214242 2024-04-19T02:03:15.864 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.864 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.864 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:15.864 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.340639+0000", 2024-04-19T02:03:15.865 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9449000000000003e-05 2024-04-19T02:03:15.865 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.865 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:15.865 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.865 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.865 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.865 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:160 1.0 e19/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'114, mlcod=19'108)", 2024-04-19T02:03:15.865 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:01.297268+0000", 2024-04-19T02:03:15.865 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 14.503336027, 2024-04-19T02:03:15.865 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0044380230000000001, 2024-04-19T02:03:15.865 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:15.865 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:15.865 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:15.866 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:15.866 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.866 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:15.866 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.297268+0000", 2024-04-19T02:03:15.866 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.866 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.866 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.866 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:15.866 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.297268+0000", 2024-04-19T02:03:15.866 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.866 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.867 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.867 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:15.867 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.297269+0000", 2024-04-19T02:03:15.867 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.61e-06 2024-04-19T02:03:15.867 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.867 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.867 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:15.867 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.297282+0000", 2024-04-19T02:03:15.867 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.279e-05 2024-04-19T02:03:15.867 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.867 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.868 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:15.868 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.297312+0000", 2024-04-19T02:03:15.868 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0015999999999999e-05 2024-04-19T02:03:15.868 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.868 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.868 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:15.868 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.297329+0000", 2024-04-19T02:03:15.868 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6843000000000001e-05 2024-04-19T02:03:15.868 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.868 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.868 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:15.868 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.297412+0000", 2024-04-19T02:03:15.868 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2646999999999997e-05 2024-04-19T02:03:15.868 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.869 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.869 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:15.869 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.297439+0000", 2024-04-19T02:03:15.869 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7259000000000001e-05 2024-04-19T02:03:15.869 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.869 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.869 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:15.869 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.301612+0000", 2024-04-19T02:03:15.869 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0041734190000000003 2024-04-19T02:03:15.869 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.869 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.869 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:15.869 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.301706+0000", 2024-04-19T02:03:15.869 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3438999999999998e-05 2024-04-19T02:03:15.870 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.870 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:15.870 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.870 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.870 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.870 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:161 1.0 e19/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'115, mlcod=19'111)", 2024-04-19T02:03:15.870 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:01.298463+0000", 2024-04-19T02:03:15.870 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 14.502141323, 2024-04-19T02:03:15.870 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0051605569999999996, 2024-04-19T02:03:15.870 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:15.870 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:15.870 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:15.870 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:15.870 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.870 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:15.871 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.298463+0000", 2024-04-19T02:03:15.871 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.871 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.871 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.871 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:15.871 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.298463+0000", 2024-04-19T02:03:15.871 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.871 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.871 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.871 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:15.871 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.298464+0000", 2024-04-19T02:03:15.871 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.72e-06 2024-04-19T02:03:15.871 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.871 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.871 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:15.871 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.298577+0000", 2024-04-19T02:03:15.872 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000113083 2024-04-19T02:03:15.872 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.872 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.872 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:15.872 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.298608+0000", 2024-04-19T02:03:15.872 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1115999999999998e-05 2024-04-19T02:03:15.872 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.872 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.872 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:15.872 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.298633+0000", 2024-04-19T02:03:15.872 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4542999999999999e-05 2024-04-19T02:03:15.872 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.872 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.872 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:15.872 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.298761+0000", 2024-04-19T02:03:15.872 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000128026 2024-04-19T02:03:15.873 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.873 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.873 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:15.873 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.298817+0000", 2024-04-19T02:03:15.873 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5495999999999999e-05 2024-04-19T02:03:15.873 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.873 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.873 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:15.873 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.303529+0000", 2024-04-19T02:03:15.873 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0047123970000000001 2024-04-19T02:03:15.873 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.873 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.873 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:15.873 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.303623+0000", 2024-04-19T02:03:15.873 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4176000000000001e-05 2024-04-19T02:03:15.873 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.873 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:15.874 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.874 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.874 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.874 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:162 1.0 e19/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'116, mlcod=19'111)", 2024-04-19T02:03:15.874 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:01.299115+0000", 2024-04-19T02:03:15.874 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 14.501488491, 2024-04-19T02:03:15.874 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0045818350000000002, 2024-04-19T02:03:15.874 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:15.874 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:15.874 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:15.874 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:15.874 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.874 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:15.874 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.299115+0000", 2024-04-19T02:03:15.874 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.875 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.875 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.875 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:15.875 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.299115+0000", 2024-04-19T02:03:15.875 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.875 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.875 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.875 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:15.875 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.299117+0000", 2024-04-19T02:03:15.875 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4270000000000001e-06 2024-04-19T02:03:15.875 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.875 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.875 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:15.875 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.299126+0000", 2024-04-19T02:03:15.875 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6630000000000005e-06 2024-04-19T02:03:15.876 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.876 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.876 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:15.876 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.299143+0000", 2024-04-19T02:03:15.876 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6793e-05 2024-04-19T02:03:15.876 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.876 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.876 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:15.876 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.299154+0000", 2024-04-19T02:03:15.876 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0273e-05 2024-04-19T02:03:15.876 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.876 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.876 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:15.876 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.300143+0000", 2024-04-19T02:03:15.877 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00098952299999999997 2024-04-19T02:03:15.877 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.877 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.877 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:15.877 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.300175+0000", 2024-04-19T02:03:15.877 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.201e-05 2024-04-19T02:03:15.877 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.877 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.877 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:15.877 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.303627+0000", 2024-04-19T02:03:15.877 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003451866 2024-04-19T02:03:15.877 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.877 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.877 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:15.877 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.303697+0000", 2024-04-19T02:03:15.877 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0279999999999998e-05 2024-04-19T02:03:15.877 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.878 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:15.878 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.878 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.878 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.878 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:163 1.0 e19/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'117, mlcod=19'112)", 2024-04-19T02:03:15.878 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:01.299976+0000", 2024-04-19T02:03:15.878 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 14.500627680999999, 2024-04-19T02:03:15.878 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0037995070000000001, 2024-04-19T02:03:15.878 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:15.878 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:15.878 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:15.878 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:15.879 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.879 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:15.879 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.299976+0000", 2024-04-19T02:03:15.879 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.879 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.879 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.879 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:15.879 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.299976+0000", 2024-04-19T02:03:15.879 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.879 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.879 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.879 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:15.880 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.299977+0000", 2024-04-19T02:03:15.880 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.17e-06 2024-04-19T02:03:15.880 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.880 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.880 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:15.880 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.299991+0000", 2024-04-19T02:03:15.880 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.324e-05 2024-04-19T02:03:15.880 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.880 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.880 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:15.880 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.300009+0000", 2024-04-19T02:03:15.880 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8490000000000001e-05 2024-04-19T02:03:15.880 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.881 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.881 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:15.881 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.300019+0000", 2024-04-19T02:03:15.881 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6430000000000007e-06 2024-04-19T02:03:15.881 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.881 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.881 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:15.881 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.300687+0000", 2024-04-19T02:03:15.881 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00066863800000000004 2024-04-19T02:03:15.881 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.882 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.882 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:15.882 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.300748+0000", 2024-04-19T02:03:15.882 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0112999999999999e-05 2024-04-19T02:03:15.882 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.882 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.882 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:15.882 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.303704+0000", 2024-04-19T02:03:15.882 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0029559970000000001 2024-04-19T02:03:15.882 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.882 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.882 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:15.883 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.303776+0000", 2024-04-19T02:03:15.883 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2216000000000001e-05 2024-04-19T02:03:15.883 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.883 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:15.883 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.883 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.883 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.883 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:179 1.0 e21/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'129, mlcod=19'125)", 2024-04-19T02:03:15.883 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:01.981445+0000", 2024-04-19T02:03:15.883 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 13.819159062000001, 2024-04-19T02:03:15.883 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0035471220000000002, 2024-04-19T02:03:15.883 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:15.884 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:15.884 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:15.884 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:15.884 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.884 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:15.884 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.981445+0000", 2024-04-19T02:03:15.884 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.884 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.884 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.884 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:15.884 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.981445+0000", 2024-04-19T02:03:15.884 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:15.885 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.885 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.885 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:15.885 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.981446+0000", 2024-04-19T02:03:15.885 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9699999999999994e-07 2024-04-19T02:03:15.885 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.885 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.885 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:15.885 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.981641+0000", 2024-04-19T02:03:15.885 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00019564499999999999 2024-04-19T02:03:15.885 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.885 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.885 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:15.886 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.981658+0000", 2024-04-19T02:03:15.886 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6296000000000001e-05 2024-04-19T02:03:15.886 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.886 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.886 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:15.886 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.981668+0000", 2024-04-19T02:03:15.886 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0346999999999999e-05 2024-04-19T02:03:15.886 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.886 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.886 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:15.886 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.982459+0000", 2024-04-19T02:03:15.887 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00079046400000000005 2024-04-19T02:03:15.887 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.887 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.887 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:15.887 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.982511+0000", 2024-04-19T02:03:15.887 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2562999999999998e-05 2024-04-19T02:03:15.887 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.887 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.887 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:15.887 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.984939+0000", 2024-04-19T02:03:15.887 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0024275049999999999 2024-04-19T02:03:15.887 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:15.888 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:15.888 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:15.888 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.984992+0000", 2024-04-19T02:03:15.888 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3304999999999999e-05 2024-04-19T02:03:15.888 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.888 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:15.888 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.888 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:15.888 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:15.888 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:15.890 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-19T02:03:15.890 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:15.891+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:15.990 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:15.991+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:16.059 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:16.059 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:16.059 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 0 2024-04-19T02:03:16.059 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:16.073 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-19T02:03:16.117 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:16.119+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:16.192 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:16.191+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:16.194 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:16.194 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:16.194 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-04-19T02:03:16.194 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-04-19T02:03:16.194 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:16.208 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-19T02:03:16.293 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:16.295+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:16.373 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:16.374 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-04-19T02:03:16.374 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-04-19T02:03:16.374 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [] 2024-04-19T02:03:16.374 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:16.387 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-19T02:03:16.394 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:16.395+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:16.495 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:16.495+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:16.510 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:16.510 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:16.510 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 0 2024-04-19T02:03:16.510 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:16.524 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-19T02:03:16.596 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:16.595+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:16.638 INFO:tasks.rados.rados.0.smithi059.stdout:84: done (1 left) 2024-04-19T02:03:16.638 INFO:tasks.rados.rados.0.smithi059.stdout:85: setattr oid 17 current snap is 2 2024-04-19T02:03:16.638 INFO:tasks.rados.rados.0.smithi059.stdout:83: finishing write tid 1 to smithi05930959-23 2024-04-19T02:03:16.638 INFO:tasks.rados.rados.0.smithi059.stdout:83: finishing write tid 2 to smithi05930959-23 2024-04-19T02:03:16.639 INFO:tasks.rados.rados.0.smithi059.stdout:83: finishing write tid 3 to smithi05930959-23 2024-04-19T02:03:16.639 INFO:tasks.rados.rados.0.smithi059.stdout:83: finishing write tid 4 to smithi05930959-23 2024-04-19T02:03:16.639 INFO:tasks.rados.rados.0.smithi059.stdout:83: oid 23 updating version 0 to 28 2024-04-19T02:03:16.639 INFO:tasks.rados.rados.0.smithi059.stdout:83: oid 23 version 28 is already newer than 27 2024-04-19T02:03:16.639 INFO:tasks.rados.rados.0.smithi059.stdout:83: oid 23 version 28 is already newer than 26 2024-04-19T02:03:16.639 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 23 v 28 (ObjNum 62 snap 2 seq_num 62) dirty exists 2024-04-19T02:03:16.639 INFO:tasks.rados.rados.0.smithi059.stdout:83: left oid 23 (ObjNum 62 snap 2 seq_num 62) 2024-04-19T02:03:16.641 INFO:tasks.rados.rados.0.smithi059.stdout:83: done (1 left) 2024-04-19T02:03:16.641 INFO:tasks.rados.rados.0.smithi059.stdout:86: read oid 22 snap 0 2024-04-19T02:03:16.641 INFO:tasks.rados.rados.0.smithi059.stdout:86: expect (ObjNum 21 snap 0 seq_num 21) 2024-04-19T02:03:16.641 INFO:tasks.rados.rados.0.smithi059.stdout:87: delete oid 12 current snap is 2 2024-04-19T02:03:16.646 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:16.646 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:16.646 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-04-19T02:03:16.647 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-04-19T02:03:16.647 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:16.653 INFO:tasks.rados.rados.0.smithi059.stdout:87: done (2 left) 2024-04-19T02:03:16.653 INFO:tasks.rados.rados.0.smithi059.stdout:88: snap_create 2024-04-19T02:03:16.657 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-19T02:03:16.663 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 17 v 29 (ObjNum 353 snap 0 seq_num 4609) dirty exists 2024-04-19T02:03:16.664 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-19T02:03:16.671 INFO:tasks.rados.rados.0.smithi059.stdout:86: expect (ObjNum 21 snap 0 seq_num 21) 2024-04-19T02:03:16.710 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:16.711+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:16.788 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:16.788 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-04-19T02:03:16.788 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-04-19T02:03:16.788 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-04-19T02:03:16.788 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.788 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:240 3.es0 3:79fe7502:::smithi05930959-12:head [stat,delete] snapc 3=[2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-19T02:03:16.788 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:16.646061+0000", 2024-04-19T02:03:16.788 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.144833463, 2024-04-19T02:03:16.788 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010522234, 2024-04-19T02:03:16.788 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:16.788 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:16.788 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:03:16.789 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:16.789 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:16.789 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:16.789 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 240 2024-04-19T02:03:16.789 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.789 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:16.789 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.789 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:16.789 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.646061+0000", 2024-04-19T02:03:16.789 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:16.789 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.789 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.789 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:16.789 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.646061+0000", 2024-04-19T02:03:16.790 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:16.790 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.790 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.790 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:16.790 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.646064+0000", 2024-04-19T02:03:16.790 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3230000000000002e-06 2024-04-19T02:03:16.790 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.790 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.790 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:16.790 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.646080+0000", 2024-04-19T02:03:16.790 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5566999999999999e-05 2024-04-19T02:03:16.790 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.790 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.790 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:16.791 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.646119+0000", 2024-04-19T02:03:16.791 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9323000000000003e-05 2024-04-19T02:03:16.791 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.791 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.791 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:16.791 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.646143+0000", 2024-04-19T02:03:16.791 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.372e-05 2024-04-19T02:03:16.791 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.791 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.791 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:16.791 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.646309+0000", 2024-04-19T02:03:16.791 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016613899999999999 2024-04-19T02:03:16.791 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.791 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.791 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op must wait for map", 2024-04-19T02:03:16.792 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.646333+0000", 2024-04-19T02:03:16.792 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4626000000000001e-05 2024-04-19T02:03:16.792 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.792 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.792 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:16.792 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.650628+0000", 2024-04-19T02:03:16.792 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0042946 2024-04-19T02:03:16.792 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.792 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.792 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:16.792 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.650698+0000", 2024-04-19T02:03:16.792 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0241999999999996e-05 2024-04-19T02:03:16.792 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.792 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.793 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:16.793 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.656583+0000", 2024-04-19T02:03:16.793 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0058846940000000002 2024-04-19T02:03:16.793 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:16.793 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:16.793 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:16.793 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.793 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.793 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "pg_update_log_missing_reply(3.es0 epoch 32/26 rep_tid 18 lcod 32'12)", 2024-04-19T02:03:16.793 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:16.652397+0000", 2024-04-19T02:03:16.793 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.13849668100000001, 2024-04-19T02:03:16.793 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016069899999999999, 2024-04-19T02:03:16.793 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:16.793 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:16.793 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:16.794 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:16.794 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.794 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:16.794 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.652397+0000", 2024-04-19T02:03:16.794 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:16.794 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.794 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.794 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:16.794 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.652397+0000", 2024-04-19T02:03:16.794 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:16.794 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.794 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.794 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:16.794 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.652399+0000", 2024-04-19T02:03:16.795 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.463e-06 2024-04-19T02:03:16.795 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.795 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.795 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:16.795 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.652410+0000", 2024-04-19T02:03:16.795 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1399999999999999e-05 2024-04-19T02:03:16.795 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.795 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.795 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:16.795 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.652438+0000", 2024-04-19T02:03:16.795 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7512999999999999e-05 2024-04-19T02:03:16.795 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.795 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.795 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:16.795 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.652456+0000", 2024-04-19T02:03:16.796 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8243000000000001e-05 2024-04-19T02:03:16.796 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.796 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.796 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:16.796 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.652535+0000", 2024-04-19T02:03:16.796 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9015000000000002e-05 2024-04-19T02:03:16.796 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.796 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.796 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:16.796 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.652558+0000", 2024-04-19T02:03:16.796 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3065e-05 2024-04-19T02:03:16.796 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:16.796 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:16.796 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:16.796 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.797 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.797 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "pg_update_log_missing_reply(3.es0 epoch 32/26 rep_tid 18 lcod 32'12)", 2024-04-19T02:03:16.797 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:16.652452+0000", 2024-04-19T02:03:16.797 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.13844208, 2024-04-19T02:03:16.797 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014832800000000001, 2024-04-19T02:03:16.797 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:16.797 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:16.797 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:16.797 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:16.797 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.797 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:16.797 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.652452+0000", 2024-04-19T02:03:16.797 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:16.797 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.797 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.798 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:16.798 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.652452+0000", 2024-04-19T02:03:16.798 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:16.798 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.798 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.798 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:16.798 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.652453+0000", 2024-04-19T02:03:16.798 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.9800000000000002e-07 2024-04-19T02:03:16.798 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.798 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.798 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:16.798 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.652457+0000", 2024-04-19T02:03:16.798 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1949999999999996e-06 2024-04-19T02:03:16.798 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.798 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.799 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:16.799 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.652468+0000", 2024-04-19T02:03:16.799 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0530000000000001e-05 2024-04-19T02:03:16.799 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.799 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.799 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:16.799 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.652474+0000", 2024-04-19T02:03:16.799 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5529999999999998e-06 2024-04-19T02:03:16.799 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.799 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.799 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:16.799 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.652582+0000", 2024-04-19T02:03:16.799 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000107805 2024-04-19T02:03:16.799 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.799 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.800 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:16.800 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.652600+0000", 2024-04-19T02:03:16.800 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8346999999999999e-05 2024-04-19T02:03:16.800 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:16.800 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:16.800 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:16.800 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.800 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.800 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "pg_update_log_missing_reply(3.es0 epoch 32/26 rep_tid 18 lcod 32'12)", 2024-04-19T02:03:16.800 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:16.653093+0000", 2024-04-19T02:03:16.800 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.13780082399999999, 2024-04-19T02:03:16.800 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012919200000000001, 2024-04-19T02:03:16.800 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:16.800 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:16.800 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:16.801 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:16.801 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.801 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:16.801 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.653093+0000", 2024-04-19T02:03:16.801 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:16.801 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.801 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.801 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:16.801 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.653093+0000", 2024-04-19T02:03:16.801 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:16.801 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.801 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.801 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:16.801 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.653095+0000", 2024-04-19T02:03:16.802 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4699999999999999e-06 2024-04-19T02:03:16.802 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.802 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.802 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:16.802 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.653103+0000", 2024-04-19T02:03:16.802 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.6300000000000004e-06 2024-04-19T02:03:16.802 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.802 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.802 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:16.802 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.653126+0000", 2024-04-19T02:03:16.802 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3050000000000001e-05 2024-04-19T02:03:16.802 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.802 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.802 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:16.803 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.653137+0000", 2024-04-19T02:03:16.803 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0907000000000001e-05 2024-04-19T02:03:16.803 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.803 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.803 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:16.803 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.653198+0000", 2024-04-19T02:03:16.803 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0368000000000002e-05 2024-04-19T02:03:16.803 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.803 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.803 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:16.803 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.653223+0000", 2024-04-19T02:03:16.803 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4766999999999999e-05 2024-04-19T02:03:16.803 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:16.803 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:16.803 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:16.804 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.804 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.804 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "pg_update_log_missing_reply(3.es0 epoch 32/26 rep_tid 18 lcod 32'12)", 2024-04-19T02:03:16.804 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:16.653321+0000", 2024-04-19T02:03:16.804 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.13757316, 2024-04-19T02:03:16.804 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00058666000000000002, 2024-04-19T02:03:16.804 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:16.804 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:16.804 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:16.804 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:16.804 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.804 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:16.804 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.653321+0000", 2024-04-19T02:03:16.804 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:16.805 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.805 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.805 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:16.805 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.653321+0000", 2024-04-19T02:03:16.805 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:16.805 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.805 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.805 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:16.805 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.653322+0000", 2024-04-19T02:03:16.805 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.54e-07 2024-04-19T02:03:16.805 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.805 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.805 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:16.806 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.653326+0000", 2024-04-19T02:03:16.806 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6829999999999999e-06 2024-04-19T02:03:16.806 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.806 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.806 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:16.806 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.653335+0000", 2024-04-19T02:03:16.806 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8559999999999996e-06 2024-04-19T02:03:16.806 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.806 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.806 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:16.806 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.653345+0000", 2024-04-19T02:03:16.806 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5100000000000004e-06 2024-04-19T02:03:16.806 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.806 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.806 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:16.807 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.653836+0000", 2024-04-19T02:03:16.807 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000490972 2024-04-19T02:03:16.807 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.807 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.807 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:16.807 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.653908+0000", 2024-04-19T02:03:16.807 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1884999999999994e-05 2024-04-19T02:03:16.807 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:16.807 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:16.807 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:16.807 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.808 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.808 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "pg_update_log_missing_reply(3.es0 epoch 32/26 rep_tid 18 lcod 32'12)", 2024-04-19T02:03:16.808 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:16.653336+0000", 2024-04-19T02:03:16.808 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.13755830899999999, 2024-04-19T02:03:16.808 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00088399099999999999, 2024-04-19T02:03:16.808 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:16.808 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:16.808 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:16.808 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:16.808 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.808 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:16.808 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.653336+0000", 2024-04-19T02:03:16.808 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:16.808 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.809 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.809 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:16.809 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.653336+0000", 2024-04-19T02:03:16.809 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:16.809 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.809 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.809 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:16.809 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.653337+0000", 2024-04-19T02:03:16.809 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.33e-06 2024-04-19T02:03:16.809 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.809 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.809 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:16.809 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.653345+0000", 2024-04-19T02:03:16.809 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9670000000000001e-06 2024-04-19T02:03:16.809 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.810 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.810 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:16.810 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.653369+0000", 2024-04-19T02:03:16.810 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4036000000000002e-05 2024-04-19T02:03:16.810 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.810 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.810 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:16.810 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.653385+0000", 2024-04-19T02:03:16.810 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5517000000000001e-05 2024-04-19T02:03:16.810 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.810 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.810 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:16.810 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.654154+0000", 2024-04-19T02:03:16.811 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00076917700000000001 2024-04-19T02:03:16.811 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.811 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.811 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:16.811 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.654220+0000", 2024-04-19T02:03:16.811 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5963999999999994e-05 2024-04-19T02:03:16.811 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:16.811 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:16.811 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:16.811 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.811 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.811 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "pg_update_log_missing_reply(3.es0 epoch 32/26 rep_tid 18 lcod 32'12)", 2024-04-19T02:03:16.811 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:16.653439+0000", 2024-04-19T02:03:16.811 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.13745505199999999, 2024-04-19T02:03:16.811 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00065010800000000004, 2024-04-19T02:03:16.811 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:16.812 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:16.812 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:16.812 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:16.812 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.812 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:16.812 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.653439+0000", 2024-04-19T02:03:16.812 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:16.812 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.812 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.812 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:16.812 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.653439+0000", 2024-04-19T02:03:16.812 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:16.812 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.812 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.812 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:16.813 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.653440+0000", 2024-04-19T02:03:16.813 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7999999999999993e-07 2024-04-19T02:03:16.813 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.813 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.813 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:16.813 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.653448+0000", 2024-04-19T02:03:16.813 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3229999999999998e-06 2024-04-19T02:03:16.813 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.813 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.813 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:16.813 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.653470+0000", 2024-04-19T02:03:16.813 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1158999999999999e-05 2024-04-19T02:03:16.813 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.813 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.814 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:16.814 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.653482+0000", 2024-04-19T02:03:16.814 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.223e-05 2024-04-19T02:03:16.814 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.814 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.814 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:16.814 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.654017+0000", 2024-04-19T02:03:16.814 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00053507299999999995 2024-04-19T02:03:16.814 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.814 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.814 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:16.814 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.654089+0000", 2024-04-19T02:03:16.814 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2342999999999995e-05 2024-04-19T02:03:16.814 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:16.815 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:16.815 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:16.815 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.815 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.815 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "pg_update_log_missing_reply(3.es0 epoch 32/26 rep_tid 18 lcod 32'12)", 2024-04-19T02:03:16.815 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:16.656313+0000", 2024-04-19T02:03:16.815 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.13458131500000001, 2024-04-19T02:03:16.815 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00027542000000000002, 2024-04-19T02:03:16.815 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:16.815 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:16.815 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:16.815 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:16.815 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.815 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:16.815 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.656313+0000", 2024-04-19T02:03:16.816 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:16.816 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.816 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.816 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:16.816 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.656313+0000", 2024-04-19T02:03:16.816 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:16.816 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.816 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.816 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:16.816 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.656315+0000", 2024-04-19T02:03:16.816 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8729999999999999e-06 2024-04-19T02:03:16.816 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.816 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.816 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:16.817 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.656326+0000", 2024-04-19T02:03:16.817 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1143e-05 2024-04-19T02:03:16.817 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.817 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.817 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:16.817 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.656357+0000", 2024-04-19T02:03:16.817 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0670000000000003e-05 2024-04-19T02:03:16.817 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.817 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.817 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:16.817 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.656375+0000", 2024-04-19T02:03:16.817 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7986999999999999e-05 2024-04-19T02:03:16.817 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.817 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.817 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:16.818 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.656458+0000", 2024-04-19T02:03:16.818 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3421e-05 2024-04-19T02:03:16.818 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:16.818 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:16.818 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:16.818 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.656588+0000", 2024-04-19T02:03:16.818 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000130326 2024-04-19T02:03:16.818 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:16.818 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:16.818 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:16.818 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:16.818 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:16.818 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:16.820 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-19T02:03:16.820 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:16.819+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:16.920 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:16.919+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:16.950 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:16.950 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:03:16.950 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-04-19T02:03:16.950 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:16.964 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-19T02:03:17.017 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T02:03:17.021 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:17.023+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:17.081 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:17.081 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:03:17.082 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-04-19T02:03:17.082 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-04-19T02:03:17.082 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:17.095 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-19T02:03:17.123 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:17.123+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:17.170 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-04-19T02:03:17.170 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T02:03:17.187 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-04-19T02:03:17.187 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T02:03:17.204 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-04-19T02:03:17.205 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T02:03:17.222 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-04-19T02:03:17.222 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T02:03:17.223 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:17.223+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:17.224 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:17.224 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-04-19T02:03:17.225 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-04-19T02:03:17.225 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-04-19T02:03:17.225 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.225 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:17.225 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:02:59.539411+0000", 2024-04-19T02:03:17.225 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 17.685493393000002, 2024-04-19T02:03:17.225 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.43600685099999997, 2024-04-19T02:03:17.225 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:17.225 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:17.225 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:17.225 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:17.225 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:17.225 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:17.226 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 1 2024-04-19T02:03:17.226 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.226 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:17.226 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.226 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:17.226 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539411+0000", 2024-04-19T02:03:17.226 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.226 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.226 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.226 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:17.226 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539411+0000", 2024-04-19T02:03:17.226 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.226 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.227 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.227 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:17.227 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539413+0000", 2024-04-19T02:03:17.227 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3970000000000001e-06 2024-04-19T02:03:17.227 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.227 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.227 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:17.227 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539429+0000", 2024-04-19T02:03:17.227 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5153e-05 2024-04-19T02:03:17.227 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.227 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.227 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:17.227 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539450+0000", 2024-04-19T02:03:17.227 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1243e-05 2024-04-19T02:03:17.227 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.228 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.228 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:17.228 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539470+0000", 2024-04-19T02:03:17.228 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0017000000000002e-05 2024-04-19T02:03:17.228 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.228 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.228 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:17.228 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539605+0000", 2024-04-19T02:03:17.228 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013489200000000001 2024-04-19T02:03:17.228 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.228 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.228 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-04-19T02:03:17.228 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539633+0000", 2024-04-19T02:03:17.228 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8113e-05 2024-04-19T02:03:17.228 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.228 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.229 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:17.229 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.972415+0000", 2024-04-19T02:03:17.229 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.43278235300000001 2024-04-19T02:03:17.229 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.229 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.229 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:17.229 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.972942+0000", 2024-04-19T02:03:17.229 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000526379 2024-04-19T02:03:17.229 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.229 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.229 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:17.229 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.973384+0000", 2024-04-19T02:03:17.229 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00044265 2024-04-19T02:03:17.229 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.230 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.230 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:17.230 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.974795+0000", 2024-04-19T02:03:17.230 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001410592 2024-04-19T02:03:17.230 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.230 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.230 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:17.230 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.975249+0000", 2024-04-19T02:03:17.230 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00045376999999999999 2024-04-19T02:03:17.230 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.230 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.230 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:17.230 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.975346+0000", 2024-04-19T02:03:17.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7916000000000002e-05 2024-04-19T02:03:17.231 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.231 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:17.231 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.975418+0000", 2024-04-19T02:03:17.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1376000000000002e-05 2024-04-19T02:03:17.231 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.231 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:17.231 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.231 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.231 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:17.231 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.034302+0000", 2024-04-19T02:03:17.231 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 17.190602345999999, 2024-04-19T02:03:17.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012683028000000001, 2024-04-19T02:03:17.231 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:17.232 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:17.232 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:17.232 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:17.232 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:17.232 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:17.232 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 35 2024-04-19T02:03:17.232 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.232 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:17.232 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.232 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:17.232 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034302+0000", 2024-04-19T02:03:17.232 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.232 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.232 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.233 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:17.233 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034302+0000", 2024-04-19T02:03:17.233 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.233 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.233 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.233 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:17.233 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034303+0000", 2024-04-19T02:03:17.233 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4e-07 2024-04-19T02:03:17.233 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.233 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.233 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:17.233 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034443+0000", 2024-04-19T02:03:17.233 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014014499999999999 2024-04-19T02:03:17.233 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.234 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.234 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:17.234 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034459+0000", 2024-04-19T02:03:17.234 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6096999999999999e-05 2024-04-19T02:03:17.234 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.234 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.234 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:17.234 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034470+0000", 2024-04-19T02:03:17.234 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0533000000000001e-05 2024-04-19T02:03:17.234 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.234 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.234 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:17.234 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.042549+0000", 2024-04-19T02:03:17.234 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0080795309999999992 2024-04-19T02:03:17.234 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.235 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.235 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:17.235 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.042789+0000", 2024-04-19T02:03:17.235 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00023983399999999999 2024-04-19T02:03:17.235 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.235 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.235 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:17.235 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.043193+0000", 2024-04-19T02:03:17.235 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00040417699999999998 2024-04-19T02:03:17.235 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.235 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.235 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:17.235 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.046637+0000", 2024-04-19T02:03:17.235 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0034438709999999998 2024-04-19T02:03:17.236 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.236 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:17.236 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.046726+0000", 2024-04-19T02:03:17.236 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8622000000000006e-05 2024-04-19T02:03:17.236 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.236 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:17.236 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.046864+0000", 2024-04-19T02:03:17.236 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000138553 2024-04-19T02:03:17.236 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.236 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:17.236 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.046985+0000", 2024-04-19T02:03:17.236 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000120825 2024-04-19T02:03:17.237 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.237 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:17.237 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.237 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.237 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.237 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:17.237 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.034507+0000", 2024-04-19T02:03:17.237 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 17.190396942, 2024-04-19T02:03:17.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012727749, 2024-04-19T02:03:17.237 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:17.237 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:17.237 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:17.237 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:17.237 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:17.237 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:17.238 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 36 2024-04-19T02:03:17.238 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.238 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:17.238 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.238 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:17.238 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034507+0000", 2024-04-19T02:03:17.238 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.238 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.238 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.238 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:17.238 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034507+0000", 2024-04-19T02:03:17.238 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.238 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.238 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.239 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:17.239 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034508+0000", 2024-04-19T02:03:17.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0640000000000001e-06 2024-04-19T02:03:17.239 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.239 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:17.239 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034516+0000", 2024-04-19T02:03:17.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8199999999999997e-06 2024-04-19T02:03:17.239 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.239 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:17.239 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034528+0000", 2024-04-19T02:03:17.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1996e-05 2024-04-19T02:03:17.239 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.240 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:17.240 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034539+0000", 2024-04-19T02:03:17.240 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.098e-05 2024-04-19T02:03:17.240 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.240 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.240 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:17.240 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.044335+0000", 2024-04-19T02:03:17.240 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0097961309999999996 2024-04-19T02:03:17.240 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.241 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.241 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:17.241 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.044476+0000", 2024-04-19T02:03:17.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014055900000000001 2024-04-19T02:03:17.241 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.241 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.241 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:17.241 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.044678+0000", 2024-04-19T02:03:17.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000202229 2024-04-19T02:03:17.241 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.241 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.241 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:17.241 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.047076+0000", 2024-04-19T02:03:17.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0023975540000000001 2024-04-19T02:03:17.241 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.241 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.242 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:17.242 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.047096+0000", 2024-04-19T02:03:17.242 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0373999999999998e-05 2024-04-19T02:03:17.242 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.242 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.242 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:17.242 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.047168+0000", 2024-04-19T02:03:17.242 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1915999999999994e-05 2024-04-19T02:03:17.242 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.242 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.242 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:17.242 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.047235+0000", 2024-04-19T02:03:17.242 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7125999999999994e-05 2024-04-19T02:03:17.242 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.243 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:17.243 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.243 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.243 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.243 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:17.243 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.034553+0000", 2024-04-19T02:03:17.243 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 17.190350991999999, 2024-04-19T02:03:17.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013521706, 2024-04-19T02:03:17.243 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:17.243 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:17.243 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:17.243 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:17.243 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:17.243 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:17.243 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 37 2024-04-19T02:03:17.244 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.244 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:17.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.244 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:17.244 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034553+0000", 2024-04-19T02:03:17.244 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.244 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.244 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:17.244 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034553+0000", 2024-04-19T02:03:17.244 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.244 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.244 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:17.244 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034554+0000", 2024-04-19T02:03:17.245 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0699999999999996e-07 2024-04-19T02:03:17.245 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.245 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.245 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:17.245 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034562+0000", 2024-04-19T02:03:17.245 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7570000000000004e-06 2024-04-19T02:03:17.245 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.245 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.245 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:17.245 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034572+0000", 2024-04-19T02:03:17.245 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6730000000000004e-06 2024-04-19T02:03:17.245 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.245 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.245 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:17.246 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.034580+0000", 2024-04-19T02:03:17.246 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5900000000000008e-06 2024-04-19T02:03:17.246 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.246 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.246 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:17.246 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.045799+0000", 2024-04-19T02:03:17.246 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.01121861 2024-04-19T02:03:17.246 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.246 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.246 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:17.246 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.045949+0000", 2024-04-19T02:03:17.246 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000150245 2024-04-19T02:03:17.246 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.246 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.246 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:17.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.046152+0000", 2024-04-19T02:03:17.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020245899999999999 2024-04-19T02:03:17.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:17.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.047253+0000", 2024-04-19T02:03:17.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001101545 2024-04-19T02:03:17.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:17.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.047800+0000", 2024-04-19T02:03:17.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00054726199999999997 2024-04-19T02:03:17.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:17.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.047952+0000", 2024-04-19T02:03:17.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015121199999999999 2024-04-19T02:03:17.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:17.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.048075+0000", 2024-04-19T02:03:17.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012354600000000001 2024-04-19T02:03:17.248 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.248 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:17.248 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.248 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:17.248 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.134096+0000", 2024-04-19T02:03:17.248 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 17.090808076999998, 2024-04-19T02:03:17.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015770572, 2024-04-19T02:03:17.249 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:17.249 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:17.249 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:17.249 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:17.249 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:17.249 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:17.249 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 87 2024-04-19T02:03:17.249 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.249 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:17.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.249 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:17.249 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.134096+0000", 2024-04-19T02:03:17.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.249 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:17.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.134096+0000", 2024-04-19T02:03:17.250 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.250 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:17.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.134097+0000", 2024-04-19T02:03:17.250 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2699999999999999e-07 2024-04-19T02:03:17.250 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:17.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.134228+0000", 2024-04-19T02:03:17.250 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013088499999999999 2024-04-19T02:03:17.250 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.251 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:17.251 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.134240+0000", 2024-04-19T02:03:17.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1667e-05 2024-04-19T02:03:17.251 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.251 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.251 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:17.251 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.134248+0000", 2024-04-19T02:03:17.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4800000000000001e-06 2024-04-19T02:03:17.251 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.251 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.251 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:17.251 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.139866+0000", 2024-04-19T02:03:17.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0056177400000000004 2024-04-19T02:03:17.252 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.252 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.252 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:17.252 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.140004+0000", 2024-04-19T02:03:17.252 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013788599999999999 2024-04-19T02:03:17.252 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.252 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.252 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:17.252 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.140207+0000", 2024-04-19T02:03:17.252 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020337099999999999 2024-04-19T02:03:17.252 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.252 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.252 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:17.252 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.142576+0000", 2024-04-19T02:03:17.252 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0023685989999999999 2024-04-19T02:03:17.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:17.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.149584+0000", 2024-04-19T02:03:17.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0070081859999999996 2024-04-19T02:03:17.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:17.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.149754+0000", 2024-04-19T02:03:17.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017020100000000001 2024-04-19T02:03:17.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:17.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.149867+0000", 2024-04-19T02:03:17.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011283 2024-04-19T02:03:17.254 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.254 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:17.254 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.254 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.254 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.254 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:17.254 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.134262+0000", 2024-04-19T02:03:17.254 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 17.090642001999999, 2024-04-19T02:03:17.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015881170999999999, 2024-04-19T02:03:17.254 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:17.254 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:17.254 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:17.254 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:17.254 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:17.254 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:17.255 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 88 2024-04-19T02:03:17.255 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.255 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:17.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:17.255 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.134262+0000", 2024-04-19T02:03:17.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.255 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:17.255 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.134262+0000", 2024-04-19T02:03:17.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.255 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:17.256 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.134263+0000", 2024-04-19T02:03:17.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8999999999999995e-07 2024-04-19T02:03:17.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:17.256 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.134270+0000", 2024-04-19T02:03:17.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.187e-06 2024-04-19T02:03:17.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:17.256 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.134280+0000", 2024-04-19T02:03:17.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2070000000000001e-06 2024-04-19T02:03:17.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:17.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.134289+0000", 2024-04-19T02:03:17.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2960000000000004e-06 2024-04-19T02:03:17.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:17.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.141715+0000", 2024-04-19T02:03:17.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0074262069999999998 2024-04-19T02:03:17.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:17.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.141839+0000", 2024-04-19T02:03:17.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012360499999999999 2024-04-19T02:03:17.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:17.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.142041+0000", 2024-04-19T02:03:17.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020266899999999999 2024-04-19T02:03:17.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:17.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.143179+0000", 2024-04-19T02:03:17.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0011378880000000001 2024-04-19T02:03:17.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:17.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.149987+0000", 2024-04-19T02:03:17.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0068079270000000001 2024-04-19T02:03:17.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:17.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.150078+0000", 2024-04-19T02:03:17.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0748999999999995e-05 2024-04-19T02:03:17.259 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.259 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:17.259 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.150144+0000", 2024-04-19T02:03:17.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5545999999999999e-05 2024-04-19T02:03:17.259 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.259 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:17.259 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.259 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.260 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:17.260 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.313453+0000", 2024-04-19T02:03:17.260 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 16.911451135, 2024-04-19T02:03:17.260 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014007538999999999, 2024-04-19T02:03:17.260 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:17.260 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:17.260 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:17.260 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:17.260 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:17.260 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:17.260 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 112 2024-04-19T02:03:17.260 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.260 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:17.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.260 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:17.261 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313453+0000", 2024-04-19T02:03:17.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.261 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.261 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:17.261 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313453+0000", 2024-04-19T02:03:17.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.261 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.261 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:17.261 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313455+0000", 2024-04-19T02:03:17.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3570000000000001e-06 2024-04-19T02:03:17.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.261 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.261 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:17.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313720+0000", 2024-04-19T02:03:17.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00026500500000000001 2024-04-19T02:03:17.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:17.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313741+0000", 2024-04-19T02:03:17.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1365999999999998e-05 2024-04-19T02:03:17.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:17.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313761+0000", 2024-04-19T02:03:17.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0100000000000001e-05 2024-04-19T02:03:17.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:17.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.322587+0000", 2024-04-19T02:03:17.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0088254990000000005 2024-04-19T02:03:17.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:17.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.322724+0000", 2024-04-19T02:03:17.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013711600000000001 2024-04-19T02:03:17.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:17.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.322929+0000", 2024-04-19T02:03:17.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020514799999999999 2024-04-19T02:03:17.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:17.264 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.327208+0000", 2024-04-19T02:03:17.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0042794469999999996 2024-04-19T02:03:17.264 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.264 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.264 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:17.264 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.327318+0000", 2024-04-19T02:03:17.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011006900000000001 2024-04-19T02:03:17.264 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.264 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.264 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:17.264 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.327401+0000", 2024-04-19T02:03:17.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2206000000000005e-05 2024-04-19T02:03:17.264 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.264 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.264 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:17.265 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.327461+0000", 2024-04-19T02:03:17.265 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0226000000000002e-05 2024-04-19T02:03:17.265 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.265 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:17.265 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.265 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.265 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.265 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:17.265 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.313793+0000", 2024-04-19T02:03:17.265 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 16.911111440999999, 2024-04-19T02:03:17.265 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016785001000000001, 2024-04-19T02:03:17.265 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:17.265 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:17.265 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:17.265 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:17.266 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:17.266 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:17.266 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 113 2024-04-19T02:03:17.266 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.266 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:17.266 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.266 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:17.266 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313793+0000", 2024-04-19T02:03:17.266 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.266 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.266 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.266 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:17.266 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313793+0000", 2024-04-19T02:03:17.266 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.266 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:17.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313795+0000", 2024-04-19T02:03:17.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7740000000000001e-06 2024-04-19T02:03:17.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:17.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313806+0000", 2024-04-19T02:03:17.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1508999999999999e-05 2024-04-19T02:03:17.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:17.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313825+0000", 2024-04-19T02:03:17.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8807000000000001e-05 2024-04-19T02:03:17.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.268 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:17.268 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313840+0000", 2024-04-19T02:03:17.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.483e-05 2024-04-19T02:03:17.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.268 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:17.268 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.323657+0000", 2024-04-19T02:03:17.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0098173779999999999 2024-04-19T02:03:17.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.268 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:17.268 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.323775+0000", 2024-04-19T02:03:17.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000117706 2024-04-19T02:03:17.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.269 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:17.269 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.323962+0000", 2024-04-19T02:03:17.269 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00018726499999999999 2024-04-19T02:03:17.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.269 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:17.269 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.328438+0000", 2024-04-19T02:03:17.269 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0044756379999999997 2024-04-19T02:03:17.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.269 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:17.269 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.330417+0000", 2024-04-19T02:03:17.269 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001978998 2024-04-19T02:03:17.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.270 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:17.270 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.330516+0000", 2024-04-19T02:03:17.270 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8752999999999995e-05 2024-04-19T02:03:17.270 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.270 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.270 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:17.270 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.330578+0000", 2024-04-19T02:03:17.270 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2342999999999996e-05 2024-04-19T02:03:17.270 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.270 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:17.270 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.270 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.270 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.270 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:17.271 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.313863+0000", 2024-04-19T02:03:17.271 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 16.911041591, 2024-04-19T02:03:17.271 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017815952999999999, 2024-04-19T02:03:17.271 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:17.271 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:17.273 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:17.273 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:17.274 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:17.274 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:17.274 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 114 2024-04-19T02:03:17.274 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.274 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:17.274 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.274 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:17.274 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313863+0000", 2024-04-19T02:03:17.274 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.274 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.274 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.274 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:17.274 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313863+0000", 2024-04-19T02:03:17.274 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.274 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:17.275 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313864+0000", 2024-04-19T02:03:17.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.403e-06 2024-04-19T02:03:17.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:17.275 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313875+0000", 2024-04-19T02:03:17.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0444e-05 2024-04-19T02:03:17.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:17.275 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313893+0000", 2024-04-19T02:03:17.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8278999999999999e-05 2024-04-19T02:03:17.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.276 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:17.276 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313906+0000", 2024-04-19T02:03:17.276 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3553999999999999e-05 2024-04-19T02:03:17.276 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.276 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.276 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:17.276 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.324841+0000", 2024-04-19T02:03:17.276 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010934442000000001 2024-04-19T02:03:17.276 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.276 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.276 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:17.276 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.325027+0000", 2024-04-19T02:03:17.276 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000185995 2024-04-19T02:03:17.276 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.276 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.277 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:17.277 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.325253+0000", 2024-04-19T02:03:17.277 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000226185 2024-04-19T02:03:17.277 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.277 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.277 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:17.277 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.328457+0000", 2024-04-19T02:03:17.277 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0032043750000000002 2024-04-19T02:03:17.277 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.277 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.277 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:17.277 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.331509+0000", 2024-04-19T02:03:17.277 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003051607 2024-04-19T02:03:17.277 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.277 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.277 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:17.278 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.331610+0000", 2024-04-19T02:03:17.278 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000101366 2024-04-19T02:03:17.278 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.278 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.278 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:17.278 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.331679+0000", 2024-04-19T02:03:17.278 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8303e-05 2024-04-19T02:03:17.278 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.278 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:17.278 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.278 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.278 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.278 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:17.278 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.313928+0000", 2024-04-19T02:03:17.278 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 16.910976317999999, 2024-04-19T02:03:17.279 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017995457999999999, 2024-04-19T02:03:17.279 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:17.279 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:17.279 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:17.279 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:17.279 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:17.279 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:17.279 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 115 2024-04-19T02:03:17.279 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.279 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:17.279 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.279 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:17.279 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313928+0000", 2024-04-19T02:03:17.279 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.280 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.280 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.280 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:17.280 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313928+0000", 2024-04-19T02:03:17.280 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.280 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.280 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.280 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:17.280 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313929+0000", 2024-04-19T02:03:17.280 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.319e-06 2024-04-19T02:03:17.280 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.280 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.280 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:17.280 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314134+0000", 2024-04-19T02:03:17.280 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020471899999999999 2024-04-19T02:03:17.281 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.281 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.281 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:17.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314157+0000", 2024-04-19T02:03:17.281 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.27e-05 2024-04-19T02:03:17.281 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.281 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.281 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:17.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314174+0000", 2024-04-19T02:03:17.281 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.694e-05 2024-04-19T02:03:17.281 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.281 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.281 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:17.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.325978+0000", 2024-04-19T02:03:17.281 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011804385000000001 2024-04-19T02:03:17.282 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.282 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.282 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:17.282 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.326092+0000", 2024-04-19T02:03:17.282 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000113916 2024-04-19T02:03:17.282 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.282 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.282 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:17.282 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.326279+0000", 2024-04-19T02:03:17.282 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000186632 2024-04-19T02:03:17.282 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.282 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.282 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:17.282 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.328477+0000", 2024-04-19T02:03:17.283 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0021981660000000001 2024-04-19T02:03:17.283 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.283 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.283 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:17.283 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.331786+0000", 2024-04-19T02:03:17.283 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003309055 2024-04-19T02:03:17.283 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.283 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.283 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:17.283 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.331864+0000", 2024-04-19T02:03:17.283 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8382999999999998e-05 2024-04-19T02:03:17.283 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.283 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.283 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:17.284 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.331923+0000", 2024-04-19T02:03:17.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9243000000000002e-05 2024-04-19T02:03:17.284 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.284 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:17.284 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.284 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.284 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.284 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:17.284 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.314202+0000", 2024-04-19T02:03:17.284 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 16.910702301000001, 2024-04-19T02:03:17.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017941301999999999, 2024-04-19T02:03:17.284 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:17.284 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:17.284 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:17.284 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:17.285 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:17.285 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:17.285 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 116 2024-04-19T02:03:17.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.285 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:17.285 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.285 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:17.285 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314202+0000", 2024-04-19T02:03:17.285 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.285 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.285 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:17.285 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314202+0000", 2024-04-19T02:03:17.285 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.286 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.286 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:17.286 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314204+0000", 2024-04-19T02:03:17.286 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7540000000000001e-06 2024-04-19T02:03:17.286 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.286 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.286 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:17.286 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314215+0000", 2024-04-19T02:03:17.286 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1345999999999999e-05 2024-04-19T02:03:17.286 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.286 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.286 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:17.286 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314233+0000", 2024-04-19T02:03:17.286 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7487e-05 2024-04-19T02:03:17.286 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.287 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.287 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:17.287 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314249+0000", 2024-04-19T02:03:17.287 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6107e-05 2024-04-19T02:03:17.287 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.287 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.287 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:17.287 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.327551+0000", 2024-04-19T02:03:17.287 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013301840000000001 2024-04-19T02:03:17.287 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.287 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.287 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:17.287 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.327676+0000", 2024-04-19T02:03:17.287 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000125716 2024-04-19T02:03:17.287 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.288 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.288 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:17.288 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.327874+0000", 2024-04-19T02:03:17.288 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00019817500000000001 2024-04-19T02:03:17.288 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.288 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.288 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:17.288 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.330619+0000", 2024-04-19T02:03:17.288 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0027450159999999999 2024-04-19T02:03:17.288 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.288 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.288 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:17.288 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.332017+0000", 2024-04-19T02:03:17.288 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0013973659999999999 2024-04-19T02:03:17.288 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.289 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.289 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:17.289 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.332090+0000", 2024-04-19T02:03:17.289 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2889e-05 2024-04-19T02:03:17.289 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.289 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.289 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:17.289 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.332143+0000", 2024-04-19T02:03:17.289 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3606000000000001e-05 2024-04-19T02:03:17.289 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.289 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:17.289 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.289 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.289 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.290 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:17.290 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.314271+0000", 2024-04-19T02:03:17.290 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 16.910633490999999, 2024-04-19T02:03:17.290 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020466839000000001, 2024-04-19T02:03:17.290 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:17.290 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:17.290 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:17.290 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:17.290 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:17.290 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:17.290 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 117 2024-04-19T02:03:17.290 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.290 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:17.290 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.291 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:17.291 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314271+0000", 2024-04-19T02:03:17.291 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.291 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.291 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.291 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:17.291 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314271+0000", 2024-04-19T02:03:17.291 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.291 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.291 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.291 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:17.291 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314272+0000", 2024-04-19T02:03:17.291 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.514e-06 2024-04-19T02:03:17.291 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.291 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.291 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:17.292 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314284+0000", 2024-04-19T02:03:17.292 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1836e-05 2024-04-19T02:03:17.292 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.292 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.292 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:17.292 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314303+0000", 2024-04-19T02:03:17.292 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8536e-05 2024-04-19T02:03:17.292 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.292 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.292 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:17.292 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314317+0000", 2024-04-19T02:03:17.292 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4209999999999999e-05 2024-04-19T02:03:17.292 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.292 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.293 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:17.293 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.328529+0000", 2024-04-19T02:03:17.293 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014211926999999999 2024-04-19T02:03:17.293 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.293 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.293 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:17.293 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.328658+0000", 2024-04-19T02:03:17.293 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000128763 2024-04-19T02:03:17.293 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.293 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.293 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:17.293 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.328864+0000", 2024-04-19T02:03:17.293 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020656099999999999 2024-04-19T02:03:17.293 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.294 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.294 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:17.294 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.331309+0000", 2024-04-19T02:03:17.294 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0024444879999999999 2024-04-19T02:03:17.294 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.294 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.294 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:17.294 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.334581+0000", 2024-04-19T02:03:17.294 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0032721450000000002 2024-04-19T02:03:17.294 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.294 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.294 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:17.294 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.334672+0000", 2024-04-19T02:03:17.294 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1267000000000006e-05 2024-04-19T02:03:17.294 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.295 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.295 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:17.295 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.334738+0000", 2024-04-19T02:03:17.295 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5592000000000005e-05 2024-04-19T02:03:17.295 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.295 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:17.295 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.295 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.295 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.295 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:17.295 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.791344+0000", 2024-04-19T02:03:17.295 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 16.433560503999999, 2024-04-19T02:03:17.295 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015697229, 2024-04-19T02:03:17.295 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:17.296 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:17.296 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:17.296 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:17.296 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:17.296 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:17.296 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 135 2024-04-19T02:03:17.296 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.296 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:17.296 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.296 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:17.296 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791344+0000", 2024-04-19T02:03:17.296 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.296 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.296 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.296 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:17.297 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791344+0000", 2024-04-19T02:03:17.297 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.297 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.297 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.297 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:17.297 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791345+0000", 2024-04-19T02:03:17.297 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7000000000000003e-07 2024-04-19T02:03:17.297 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.297 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.297 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:17.297 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791351+0000", 2024-04-19T02:03:17.297 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8100000000000003e-06 2024-04-19T02:03:17.297 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.297 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.297 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:17.297 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791360+0000", 2024-04-19T02:03:17.298 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3170000000000008e-06 2024-04-19T02:03:17.298 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.298 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.298 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:17.298 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791368+0000", 2024-04-19T02:03:17.298 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5800000000000003e-06 2024-04-19T02:03:17.298 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.298 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.298 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:17.298 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.800383+0000", 2024-04-19T02:03:17.298 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0090151569999999993 2024-04-19T02:03:17.298 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.298 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.298 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:17.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.800622+0000", 2024-04-19T02:03:17.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000239564 2024-04-19T02:03:17.299 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.299 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:17.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.800969+0000", 2024-04-19T02:03:17.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00034647500000000001 2024-04-19T02:03:17.299 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.299 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:17.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.803767+0000", 2024-04-19T02:03:17.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002798488 2024-04-19T02:03:17.299 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.299 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:17.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.806778+0000", 2024-04-19T02:03:17.300 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0030110430000000001 2024-04-19T02:03:17.300 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.300 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.300 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:17.300 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.806913+0000", 2024-04-19T02:03:17.300 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000134332 2024-04-19T02:03:17.300 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.300 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.300 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:17.300 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.807041+0000", 2024-04-19T02:03:17.300 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000128493 2024-04-19T02:03:17.300 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.300 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:17.300 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.300 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.301 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.301 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:17.301 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.791379+0000", 2024-04-19T02:03:17.301 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 16.433525850999999, 2024-04-19T02:03:17.301 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.01942953, 2024-04-19T02:03:17.301 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:17.301 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:17.301 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:17.301 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:17.301 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:17.301 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:17.302 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 136 2024-04-19T02:03:17.302 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.302 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:17.302 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.302 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:17.302 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791379+0000", 2024-04-19T02:03:17.302 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.302 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.302 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.302 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:17.302 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791379+0000", 2024-04-19T02:03:17.302 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.302 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.302 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.303 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:17.303 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791379+0000", 2024-04-19T02:03:17.303 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4000000000000001e-07 2024-04-19T02:03:17.303 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.303 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.303 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:17.303 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791387+0000", 2024-04-19T02:03:17.303 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9160000000000001e-06 2024-04-19T02:03:17.303 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.303 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.303 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:17.303 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791396+0000", 2024-04-19T02:03:17.303 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2939999999999994e-06 2024-04-19T02:03:17.303 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.303 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.304 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:17.304 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791404+0000", 2024-04-19T02:03:17.304 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6329999999999996e-06 2024-04-19T02:03:17.304 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.304 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.304 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:17.304 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.802771+0000", 2024-04-19T02:03:17.304 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011366619 2024-04-19T02:03:17.304 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.304 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.304 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:17.304 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.802918+0000", 2024-04-19T02:03:17.304 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014683900000000001 2024-04-19T02:03:17.304 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.304 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.304 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:17.305 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.803129+0000", 2024-04-19T02:03:17.305 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021161899999999999 2024-04-19T02:03:17.305 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.305 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.305 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:17.305 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.806732+0000", 2024-04-19T02:03:17.305 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0036026679999999998 2024-04-19T02:03:17.305 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.305 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.305 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:17.305 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.810544+0000", 2024-04-19T02:03:17.305 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0038118470000000002 2024-04-19T02:03:17.305 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.305 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.306 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:17.306 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.810676+0000", 2024-04-19T02:03:17.306 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013222900000000001 2024-04-19T02:03:17.306 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.306 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.306 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:17.306 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.810808+0000", 2024-04-19T02:03:17.306 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013212599999999999 2024-04-19T02:03:17.306 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.306 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:17.306 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.306 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.306 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.306 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:17.307 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.791415+0000", 2024-04-19T02:03:17.307 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 16.433489180999999, 2024-04-19T02:03:17.307 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019711117, 2024-04-19T02:03:17.307 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:17.307 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:17.307 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:17.307 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:17.307 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:17.307 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:17.307 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 137 2024-04-19T02:03:17.307 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.307 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:17.307 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.307 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:17.307 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791415+0000", 2024-04-19T02:03:17.308 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.308 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.308 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.308 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:17.308 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791415+0000", 2024-04-19T02:03:17.308 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.308 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.308 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.308 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:17.308 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791416+0000", 2024-04-19T02:03:17.308 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.23e-07 2024-04-19T02:03:17.308 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.308 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.308 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:17.308 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791584+0000", 2024-04-19T02:03:17.309 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016797599999999999 2024-04-19T02:03:17.309 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.309 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.309 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:17.309 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791600+0000", 2024-04-19T02:03:17.309 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5945999999999999e-05 2024-04-19T02:03:17.309 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.309 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.309 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:17.309 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791610+0000", 2024-04-19T02:03:17.309 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.01e-05 2024-04-19T02:03:17.309 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.309 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.309 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:17.309 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.803873+0000", 2024-04-19T02:03:17.310 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012262848999999999 2024-04-19T02:03:17.310 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.310 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.310 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:17.310 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.803995+0000", 2024-04-19T02:03:17.310 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012215899999999999 2024-04-19T02:03:17.310 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.310 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.310 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:17.310 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.804217+0000", 2024-04-19T02:03:17.310 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000222201 2024-04-19T02:03:17.310 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.310 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.310 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:17.311 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.808399+0000", 2024-04-19T02:03:17.311 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0041816910000000004 2024-04-19T02:03:17.311 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.311 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.311 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:17.311 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.810864+0000", 2024-04-19T02:03:17.311 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0024651650000000001 2024-04-19T02:03:17.311 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.311 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.311 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:17.311 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.811011+0000", 2024-04-19T02:03:17.311 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014738199999999999 2024-04-19T02:03:17.311 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.311 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.311 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:17.312 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.811126+0000", 2024-04-19T02:03:17.312 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000114925 2024-04-19T02:03:17.312 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.312 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:17.312 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.312 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.312 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.312 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:17.312 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.791625+0000", 2024-04-19T02:03:17.312 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 16.433279045999999, 2024-04-19T02:03:17.312 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019753863999999999, 2024-04-19T02:03:17.312 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:17.312 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:17.313 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:17.313 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:17.313 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:17.313 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:17.313 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 138 2024-04-19T02:03:17.313 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.313 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:17.313 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.313 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:17.313 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791625+0000", 2024-04-19T02:03:17.313 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.313 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.313 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.313 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:17.314 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791625+0000", 2024-04-19T02:03:17.314 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.314 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.314 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.314 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:17.314 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791626+0000", 2024-04-19T02:03:17.314 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8599999999999996e-07 2024-04-19T02:03:17.314 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.314 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.314 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:17.314 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791632+0000", 2024-04-19T02:03:17.314 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7200000000000003e-06 2024-04-19T02:03:17.314 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.314 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.314 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:17.315 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791641+0000", 2024-04-19T02:03:17.315 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4299999999999995e-06 2024-04-19T02:03:17.315 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.315 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.315 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:17.315 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791649+0000", 2024-04-19T02:03:17.315 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8129999999999996e-06 2024-04-19T02:03:17.315 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.315 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.315 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:17.315 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.805144+0000", 2024-04-19T02:03:17.315 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013494920000000001 2024-04-19T02:03:17.315 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.315 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.315 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:17.316 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.805387+0000", 2024-04-19T02:03:17.316 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000243188 2024-04-19T02:03:17.316 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.316 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.316 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:17.316 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.805776+0000", 2024-04-19T02:03:17.316 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00038853999999999999 2024-04-19T02:03:17.316 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.316 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.316 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:17.316 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.808536+0000", 2024-04-19T02:03:17.316 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0027598760000000001 2024-04-19T02:03:17.316 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.316 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.316 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:17.317 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.811151+0000", 2024-04-19T02:03:17.317 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0026147560000000002 2024-04-19T02:03:17.317 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.317 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.317 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:17.317 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.811275+0000", 2024-04-19T02:03:17.317 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012435199999999999 2024-04-19T02:03:17.317 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.317 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.317 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:17.317 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.811379+0000", 2024-04-19T02:03:17.317 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000104283 2024-04-19T02:03:17.317 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.317 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:17.318 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.318 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.318 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.318 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:17.318 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.791660+0000", 2024-04-19T02:03:17.318 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 16.43324402, 2024-04-19T02:03:17.318 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021026688000000002, 2024-04-19T02:03:17.318 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:17.318 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:17.318 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:17.318 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:17.318 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:17.318 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:17.319 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 139 2024-04-19T02:03:17.319 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.319 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:17.319 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.319 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:17.319 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791660+0000", 2024-04-19T02:03:17.319 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.319 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.319 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.319 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:17.319 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791660+0000", 2024-04-19T02:03:17.319 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.319 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.319 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.319 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:17.320 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791661+0000", 2024-04-19T02:03:17.320 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2699999999999999e-07 2024-04-19T02:03:17.320 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.320 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.320 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:17.320 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791666+0000", 2024-04-19T02:03:17.320 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4160000000000003e-06 2024-04-19T02:03:17.320 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.320 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.320 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:17.320 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791676+0000", 2024-04-19T02:03:17.320 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1169999999999993e-06 2024-04-19T02:03:17.320 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.320 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.321 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:17.321 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791683+0000", 2024-04-19T02:03:17.321 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8369999999999997e-06 2024-04-19T02:03:17.321 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.321 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.321 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:17.321 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.807160+0000", 2024-04-19T02:03:17.321 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.01547656 2024-04-19T02:03:17.321 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.321 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.321 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:17.321 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.807408+0000", 2024-04-19T02:03:17.321 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000248485 2024-04-19T02:03:17.321 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.321 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.322 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:17.322 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.807622+0000", 2024-04-19T02:03:17.322 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021381199999999999 2024-04-19T02:03:17.322 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.322 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.322 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:17.322 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812397+0000", 2024-04-19T02:03:17.322 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0047750229999999998 2024-04-19T02:03:17.322 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.322 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.322 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:17.322 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812434+0000", 2024-04-19T02:03:17.322 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6869000000000002e-05 2024-04-19T02:03:17.322 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.323 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.323 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:17.323 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812571+0000", 2024-04-19T02:03:17.323 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013649199999999999 2024-04-19T02:03:17.323 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.323 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.323 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:17.323 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812687+0000", 2024-04-19T02:03:17.323 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011635 2024-04-19T02:03:17.323 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.323 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:17.323 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.323 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.323 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.324 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:17.324 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.791694+0000", 2024-04-19T02:03:17.324 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 16.4332101, 2024-04-19T02:03:17.324 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021303507999999999, 2024-04-19T02:03:17.324 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:17.324 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:17.324 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:17.324 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:17.324 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:17.324 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:17.324 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 140 2024-04-19T02:03:17.324 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.324 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:17.324 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.324 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:17.324 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791694+0000", 2024-04-19T02:03:17.325 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.325 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.325 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.325 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:17.325 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791694+0000", 2024-04-19T02:03:17.325 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.325 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.325 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.325 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:17.325 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791695+0000", 2024-04-19T02:03:17.325 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2699999999999999e-07 2024-04-19T02:03:17.325 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.325 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.325 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:17.325 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791829+0000", 2024-04-19T02:03:17.326 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013436500000000001 2024-04-19T02:03:17.326 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.326 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.326 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:17.326 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791841+0000", 2024-04-19T02:03:17.326 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1707e-05 2024-04-19T02:03:17.326 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.326 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.326 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:17.326 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791850+0000", 2024-04-19T02:03:17.326 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5799999999999992e-06 2024-04-19T02:03:17.326 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.326 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.326 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:17.326 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.808635+0000", 2024-04-19T02:03:17.327 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016785133000000001 2024-04-19T02:03:17.327 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.327 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.327 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:17.327 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.808862+0000", 2024-04-19T02:03:17.327 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022693799999999999 2024-04-19T02:03:17.327 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.327 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.327 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:17.327 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.809242+0000", 2024-04-19T02:03:17.327 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00038053400000000003 2024-04-19T02:03:17.327 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.327 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.327 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:17.327 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812709+0000", 2024-04-19T02:03:17.328 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0034666929999999999 2024-04-19T02:03:17.328 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.328 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.328 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:17.328 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812796+0000", 2024-04-19T02:03:17.328 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6877000000000002e-05 2024-04-19T02:03:17.328 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.328 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.328 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:17.328 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812910+0000", 2024-04-19T02:03:17.328 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000113929 2024-04-19T02:03:17.328 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.328 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.328 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:17.328 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812998+0000", 2024-04-19T02:03:17.328 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8024999999999998e-05 2024-04-19T02:03:17.329 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.329 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:17.329 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.329 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.329 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.329 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:17.329 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.791863+0000", 2024-04-19T02:03:17.329 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 16.433041290999999, 2024-04-19T02:03:17.329 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021620092, 2024-04-19T02:03:17.329 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:17.329 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:17.329 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:17.329 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:17.329 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:17.329 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:17.330 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 141 2024-04-19T02:03:17.330 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.330 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:17.330 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.330 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:17.330 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791863+0000", 2024-04-19T02:03:17.330 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.330 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.330 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.330 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:17.330 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791863+0000", 2024-04-19T02:03:17.330 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.330 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.330 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.330 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:17.331 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791864+0000", 2024-04-19T02:03:17.331 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0599999999999999e-07 2024-04-19T02:03:17.331 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.331 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.331 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:17.331 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791871+0000", 2024-04-19T02:03:17.331 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1040000000000001e-06 2024-04-19T02:03:17.331 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.331 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.331 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:17.331 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791881+0000", 2024-04-19T02:03:17.331 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4329999999999993e-06 2024-04-19T02:03:17.331 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.331 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.332 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:17.332 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791888+0000", 2024-04-19T02:03:17.332 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7360000000000001e-06 2024-04-19T02:03:17.332 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.332 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.332 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:17.332 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.811476+0000", 2024-04-19T02:03:17.332 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019588186 2024-04-19T02:03:17.332 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.332 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.332 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:17.332 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.811595+0000", 2024-04-19T02:03:17.332 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011901200000000001 2024-04-19T02:03:17.332 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.333 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.333 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:17.333 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.811793+0000", 2024-04-19T02:03:17.333 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000197958 2024-04-19T02:03:17.333 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.333 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.333 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:17.333 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.813066+0000", 2024-04-19T02:03:17.333 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001272884 2024-04-19T02:03:17.333 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.333 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.333 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:17.333 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.813356+0000", 2024-04-19T02:03:17.333 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028986399999999997 2024-04-19T02:03:17.333 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.334 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.334 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:17.334 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.813430+0000", 2024-04-19T02:03:17.334 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3716999999999999e-05 2024-04-19T02:03:17.334 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.334 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.334 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:17.334 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.813483+0000", 2024-04-19T02:03:17.334 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3291999999999997e-05 2024-04-19T02:03:17.334 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.334 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:17.334 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.334 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.334 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.334 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:17.335 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:01.293321+0000", 2024-04-19T02:03:17.335 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 15.931583819, 2024-04-19T02:03:17.335 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011980067, 2024-04-19T02:03:17.335 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:17.335 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:17.335 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:17.335 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:17.335 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:17.335 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:17.335 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 165 2024-04-19T02:03:17.335 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.335 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:17.335 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.335 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:17.336 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:01.293321+0000", 2024-04-19T02:03:17.336 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.336 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.336 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.336 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:17.336 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:01.293321+0000", 2024-04-19T02:03:17.336 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:17.336 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.336 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.336 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:17.336 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:01.293321+0000", 2024-04-19T02:03:17.336 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4199999999999996e-07 2024-04-19T02:03:17.336 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.336 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.336 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:17.336 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:01.293327+0000", 2024-04-19T02:03:17.337 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7100000000000004e-06 2024-04-19T02:03:17.337 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.337 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.337 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:17.337 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:01.293332+0000", 2024-04-19T02:03:17.337 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3979999999999998e-06 2024-04-19T02:03:17.337 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.337 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.337 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:17.337 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:01.293336+0000", 2024-04-19T02:03:17.337 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9530000000000001e-06 2024-04-19T02:03:17.337 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.337 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.337 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:17.337 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:01.301596+0000", 2024-04-19T02:03:17.338 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0082593649999999994 2024-04-19T02:03:17.338 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.338 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.338 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:17.338 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:01.301713+0000", 2024-04-19T02:03:17.338 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000117846 2024-04-19T02:03:17.338 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.338 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.338 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:17.338 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:01.301886+0000", 2024-04-19T02:03:17.338 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017283899999999999 2024-04-19T02:03:17.338 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.338 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.338 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:17.338 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:01.302616+0000", 2024-04-19T02:03:17.339 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00072964899999999997 2024-04-19T02:03:17.339 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.339 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.339 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:17.339 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:01.305249+0000", 2024-04-19T02:03:17.339 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0026333279999999999 2024-04-19T02:03:17.339 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.339 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.339 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:17.339 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:01.305275+0000", 2024-04-19T02:03:17.339 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6055999999999999e-05 2024-04-19T02:03:17.339 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:17.339 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:17.339 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:17.339 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:01.305301+0000", 2024-04-19T02:03:17.340 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5381e-05 2024-04-19T02:03:17.340 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.340 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:17.340 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.340 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:17.340 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:17.340 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:17.341 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-04-19T02:03:17.341 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T02:03:17.342 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-04-19T02:03:17.342 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T02:03:17.343 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-19T02:03:17.343 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-04-19T02:03:17.343 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T02:03:17.344 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-04-19T02:03:17.344 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T02:03:17.344 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-04-19T02:03:17.344 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T02:03:17.345 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:17.343+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:17.353 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-04-19T02:03:17.353 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T02:03:17.376 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-04-19T02:03:17.377 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T02:03:17.397 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-04-19T02:03:17.398 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T02:03:17.462 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:17.463+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:17.472 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:17.473 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:17.486 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-19T02:03:17.543 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:17.543+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:17.609 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:17.609 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:17.622 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-19T02:03:17.636 INFO:teuthology.orchestra.run.smithi059.stderr:nodeep-scrub is unset 2024-04-19T02:03:17.644 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:17.643+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:17.645 INFO:tasks.rados.rados.0.smithi059.stdout:85: done (2 left) 2024-04-19T02:03:17.645 INFO:tasks.rados.rados.0.smithi059.stdout:86: done (1 left) 2024-04-19T02:03:17.645 INFO:tasks.rados.rados.0.smithi059.stdout:88: done (0 left) 2024-04-19T02:03:17.645 INFO:tasks.rados.rados.0.smithi059.stdout:89: copy_from oid 14 from oid 30 current snap is 3 2024-04-19T02:03:17.645 INFO:tasks.rados.rados.0.smithi059.stdout:90: delete oid 9 current snap is 3 2024-04-19T02:03:17.655 INFO:tasks.rados.rados.0.smithi059.stdout:90: done (1 left) 2024-04-19T02:03:17.655 INFO:tasks.rados.rados.0.smithi059.stdout:91: rollback oid 40 current snap is 3 2024-04-19T02:03:17.656 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 40 to 2 2024-04-19T02:03:17.656 INFO:tasks.rados.rados.0.smithi059.stdout:92: snap_remove snap 0 2024-04-19T02:03:17.745 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:17.747+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:17.750 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:17.750 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:17.764 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-19T02:03:17.846 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:17.847+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:17.890 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:17.890 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:17.902 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-19T02:03:17.947 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:17.947+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:18.025 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:18.025 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:18.040 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-19T02:03:18.048 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:18.047+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:18.149 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:18.147+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:18.160 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:18.160 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:18.174 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-19T02:03:18.250 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:18.251+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:18.291 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:18.291 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:18.305 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-19T02:03:18.351 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:18.351+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:18.421 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:18.421 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:18.436 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-19T02:03:18.452 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:18.451+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:18.553 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:18.555+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:18.553 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:18.553 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:18.567 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-19T02:03:18.654 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:18.655+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:18.659 INFO:tasks.rados.rados.0.smithi059.stdout:92: done (2 left) 2024-04-19T02:03:18.659 INFO:tasks.rados.rados.0.smithi059.stdout:93: read oid 25 snap -1 2024-04-19T02:03:18.659 INFO:tasks.rados.rados.0.smithi059.stdout:93: expect (ObjNum 52 snap 0 seq_num 52) 2024-04-19T02:03:18.659 INFO:tasks.rados.rados.0.smithi059.stdout:94: copy_from oid 24 from oid 43 current snap is 3 2024-04-19T02:03:18.659 INFO:tasks.rados.rados.0.smithi059.stdout:95: read oid 12 snap 2 2024-04-19T02:03:18.659 INFO:tasks.rados.rados.0.smithi059.stdout:95: expect deleted 2024-04-19T02:03:18.659 INFO:tasks.rados.rados.0.smithi059.stdout:96: copy_from oid 34 from oid 29 current snap is 3 2024-04-19T02:03:18.660 INFO:tasks.rados.rados.0.smithi059.stdout:97: read oid 16 snap -1 2024-04-19T02:03:18.660 INFO:tasks.rados.rados.0.smithi059.stdout:97: expect (ObjNum 15 snap 0 seq_num 15) 2024-04-19T02:03:18.660 INFO:tasks.rados.rados.0.smithi059.stdout:98: rollback oid 21 current snap is 3 2024-04-19T02:03:18.660 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 21 to 2 2024-04-19T02:03:18.660 INFO:tasks.rados.rados.0.smithi059.stdout:99: delete oid 27 current snap is 3 2024-04-19T02:03:18.660 INFO:tasks.rados.rados.0.smithi059.stdout:89: finishing copy_from racing read to smithi05930959-14 2024-04-19T02:03:18.660 INFO:tasks.rados.rados.0.smithi059.stdout:91: finishing rollback tid 0 to smithi05930959-40 2024-04-19T02:03:18.660 INFO:tasks.rados.rados.0.smithi059.stdout:89: finishing copy_from to smithi05930959-14 2024-04-19T02:03:18.660 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 14 v 11 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-19T02:03:18.660 INFO:tasks.rados.rados.0.smithi059.stdout:91: finishing rollback tid 1 to smithi05930959-40 2024-04-19T02:03:18.660 INFO:tasks.rados.rados.0.smithi059.stdout:91: finishing rollback tid 2 to smithi05930959-40 2024-04-19T02:03:18.660 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 40 v 18 (ObjNum 60 snap 1 seq_num 60) dirty exists 2024-04-19T02:03:18.680 INFO:tasks.rados.rados.0.smithi059.stdout:97: expect (ObjNum 15 snap 0 seq_num 15) 2024-04-19T02:03:18.699 INFO:tasks.rados.rados.0.smithi059.stdout:98: finishing rollback tid 0 to smithi05930959-21 2024-04-19T02:03:18.699 INFO:tasks.rados.rados.0.smithi059.stdout:94: finishing copy_from racing read to smithi05930959-24 2024-04-19T02:03:18.701 INFO:tasks.rados.rados.0.smithi059.stdout:98: finishing rollback tid 1 to smithi05930959-21 2024-04-19T02:03:18.701 INFO:tasks.rados.rados.0.smithi059.stdout:98: finishing rollback tid 2 to smithi05930959-21 2024-04-19T02:03:18.701 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 21 v 37 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-19T02:03:18.701 INFO:tasks.rados.rados.0.smithi059.stdout:89: done (8 left) 2024-04-19T02:03:18.701 INFO:tasks.rados.rados.0.smithi059.stdout:91: done (7 left) 2024-04-19T02:03:18.701 INFO:tasks.rados.rados.0.smithi059.stdout:95: done (6 left) 2024-04-19T02:03:18.702 INFO:tasks.rados.rados.0.smithi059.stdout:97: done (5 left) 2024-04-19T02:03:18.702 INFO:tasks.rados.rados.0.smithi059.stdout:98: done (4 left) 2024-04-19T02:03:18.702 INFO:tasks.rados.rados.0.smithi059.stdout:99: done (3 left) 2024-04-19T02:03:18.702 INFO:tasks.rados.rados.0.smithi059.stdout:100: setattr oid 46 current snap is 3 2024-04-19T02:03:18.704 INFO:tasks.rados.rados.0.smithi059.stdout:101: read oid 6 snap -1 2024-04-19T02:03:18.704 INFO:tasks.rados.rados.0.smithi059.stdout:101: expect (ObjNum 5 snap 0 seq_num 5) 2024-04-19T02:03:18.704 INFO:tasks.rados.rados.0.smithi059.stdout:102: snap_remove snap 2 2024-04-19T02:03:18.728 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:18.728 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:18.742 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-19T02:03:18.786 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:18.787+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:18.857 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:18.859+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:18.910 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:18.911 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:18.926 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-19T02:03:18.957 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:18.959+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:19.053 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:19.054 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:19.058 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:19.059+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:19.068 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-19T02:03:19.159 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:19.159+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:19.194 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:19.195 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:19.209 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-19T02:03:19.298 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:19.299+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:19.361 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:19.363+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:19.374 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:19.374 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:19.388 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-19T02:03:19.418 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T02:03:19.474 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:19.475+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:19.552 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:19.552 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:19.563 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:19.563+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:19.568 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-19T02:03:19.587 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-04-19T02:03:19.587 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T02:03:19.605 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-04-19T02:03:19.605 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T02:03:19.623 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-04-19T02:03:19.623 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T02:03:19.641 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-04-19T02:03:19.641 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T02:03:19.658 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-19T02:03:19.659 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-19T02:03:19.659 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-04-19T02:03:19.659 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T02:03:19.660 INFO:tasks.rados.rados.0.smithi059.stdout:102: done (5 left) 2024-04-19T02:03:19.660 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 49 current snap is 3 2024-04-19T02:03:19.660 INFO:tasks.rados.rados.0.smithi059.stdout:103: seq_num 66 ranges {819200=737280,1556480=376832} 2024-04-19T02:03:19.665 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:03:19.663+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:03:19.666 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-19T02:03:19.669 INFO:tasks.rados.rados.0.smithi059.stdout:103: writing smithi05930959-49 from 819200 to 1556480 tid 1 2024-04-19T02:03:19.673 INFO:tasks.rados.rados.0.smithi059.stdout:103: writing smithi05930959-49 from 1556480 to 1933312 tid 2 2024-04-19T02:03:19.673 INFO:tasks.rados.rados.0.smithi059.stdout:104: rollback oid 42 current snap is 3 2024-04-19T02:03:19.677 INFO:tasks.rados.rados.0.smithi059.stdout:93: expect (ObjNum 52 snap 0 seq_num 52) 2024-04-19T02:03:19.678 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-04-19T02:03:19.678 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T02:03:19.705 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-04-19T02:03:19.706 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T02:03:19.721 INFO:tasks.rados.rados.0.smithi059.stdout:96: finishing copy_from racing read to smithi05930959-34 2024-04-19T02:03:19.721 INFO:tasks.rados.rados.0.smithi059.stdout:94: finishing copy_from to smithi05930959-24 2024-04-19T02:03:19.721 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 24 v 4 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-19T02:03:19.721 INFO:tasks.rados.rados.0.smithi059.stdout:96: finishing copy_from to smithi05930959-34 2024-04-19T02:03:19.721 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 34 v 4 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-19T02:03:19.721 INFO:tasks.rados.rados.0.smithi059.stdout:101: expect (ObjNum 5 snap 0 seq_num 5) 2024-04-19T02:03:19.724 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 46 v 6 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-19T02:03:19.724 INFO:tasks.rados.rados.0.smithi059.stdout:103: finishing write tid 1 to smithi05930959-49 2024-04-19T02:03:19.724 INFO:tasks.rados.rados.0.smithi059.stdout:103: finishing write tid 2 to smithi05930959-49 2024-04-19T02:03:19.724 INFO:tasks.rados.rados.0.smithi059.stdout:93: done (6 left) 2024-04-19T02:03:19.724 INFO:tasks.rados.rados.0.smithi059.stdout:94: done (5 left) 2024-04-19T02:03:19.724 INFO:tasks.rados.rados.0.smithi059.stdout:96: done (4 left) 2024-04-19T02:03:19.725 INFO:tasks.rados.rados.0.smithi059.stdout:100: done (3 left) 2024-04-19T02:03:19.725 INFO:tasks.rados.rados.0.smithi059.stdout:101: done (2 left) 2024-04-19T02:03:19.725 INFO:tasks.rados.rados.0.smithi059.stdout:104: done (1 left) 2024-04-19T02:03:19.725 INFO:tasks.rados.rados.0.smithi059.stdout:105: read oid 43 snap -1 2024-04-19T02:03:19.725 INFO:tasks.rados.rados.0.smithi059.stdout:105: expect (ObjNum 42 snap 0 seq_num 42) 2024-04-19T02:03:19.725 INFO:tasks.rados.rados.0.smithi059.stdout:106: rollback oid 9 current snap is 3 2024-04-19T02:03:19.725 INFO:tasks.rados.rados.0.smithi059.stdout:106: done (2 left) 2024-04-19T02:03:19.725 INFO:tasks.rados.rados.0.smithi059.stdout:107: read oid 19 snap -1 2024-04-19T02:03:19.726 INFO:tasks.rados.rados.0.smithi059.stdout:107: expect (ObjNum 18 snap 0 seq_num 18) 2024-04-19T02:03:19.726 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 15 current snap is 3 2024-04-19T02:03:19.726 INFO:tasks.rados.rados.0.smithi059.stdout:108: seq_num 67 ranges {245760=753664,999424=376832} 2024-04-19T02:03:19.729 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:19.729 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:03:19.729 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-04-19T02:03:19.729 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:19.730 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-04-19T02:03:19.730 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T02:03:19.733 INFO:tasks.rados.rados.0.smithi059.stdout:108: writing smithi05930959-15 from 245760 to 999424 tid 1 2024-04-19T02:03:19.734 INFO:tasks.rados.rados.0.smithi059.stdout:108: writing smithi05930959-15 from 999424 to 1376256 tid 2 2024-04-19T02:03:19.734 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 41 current snap is 3 2024-04-19T02:03:19.734 INFO:tasks.rados.rados.0.smithi059.stdout:103: finishing write tid 3 to smithi05930959-49 2024-04-19T02:03:19.734 INFO:tasks.rados.rados.0.smithi059.stdout:103: finishing write tid 4 to smithi05930959-49 2024-04-19T02:03:19.734 INFO:tasks.rados.rados.0.smithi059.stdout:103: oid 49 updating version 0 to 16 2024-04-19T02:03:19.734 INFO:tasks.rados.rados.0.smithi059.stdout:103: oid 49 version 16 is already newer than 15 2024-04-19T02:03:19.735 INFO:tasks.rados.rados.0.smithi059.stdout:103: oid 49 version 16 is already newer than 14 2024-04-19T02:03:19.735 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 49 v 16 (ObjNum 66 snap 3 seq_num 66) dirty exists 2024-04-19T02:03:19.735 INFO:tasks.rados.rados.0.smithi059.stdout:103: left oid 49 (ObjNum 66 snap 3 seq_num 66) 2024-04-19T02:03:19.735 INFO:tasks.rados.rados.0.smithi059.stdout:109: seq_num 68 ranges {688128=770048,1458176=376832} 2024-04-19T02:03:19.744 INFO:tasks.rados.rados.0.smithi059.stdout:109: writing smithi05930959-41 from 688128 to 1458176 tid 1 2024-04-19T02:03:19.747 INFO:tasks.rados.rados.0.smithi059.stdout:109: writing smithi05930959-41 from 1458176 to 1835008 tid 2 2024-04-19T02:03:19.747 INFO:tasks.rados.rados.0.smithi059.stdout:103: done (4 left) 2024-04-19T02:03:19.748 INFO:tasks.rados.rados.0.smithi059.stdout:110: rollback oid 5 current snap is 3 2024-04-19T02:03:19.748 INFO:tasks.rados.rados.0.smithi059.stdout:110: done (4 left) 2024-04-19T02:03:19.748 INFO:tasks.rados.rados.0.smithi059.stdout:111: setattr oid 46 current snap is 3 2024-04-19T02:03:19.748 INFO:tasks.rados.rados.0.smithi059.stdout:105: expect (ObjNum 42 snap 0 seq_num 42) 2024-04-19T02:03:19.749 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-19T02:03:19.761 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-04-19T02:03:19.762 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T02:03:19.764 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:19.763+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:19.770 INFO:tasks.rados.rados.0.smithi059.stdout:107: expect (ObjNum 18 snap 0 seq_num 18) 2024-04-19T02:03:19.779 INFO:tasks.rados.rados.0.smithi059.stdout:108: finishing write tid 1 to smithi05930959-15 2024-04-19T02:03:19.779 INFO:tasks.rados.rados.0.smithi059.stdout:108: finishing write tid 2 to smithi05930959-15 2024-04-19T02:03:19.779 INFO:tasks.rados.rados.0.smithi059.stdout:108: finishing write tid 3 to smithi05930959-15 2024-04-19T02:03:19.779 INFO:tasks.rados.rados.0.smithi059.stdout:108: finishing write tid 4 to smithi05930959-15 2024-04-19T02:03:19.779 INFO:tasks.rados.rados.0.smithi059.stdout:108: oid 15 updating version 0 to 15 2024-04-19T02:03:19.779 INFO:tasks.rados.rados.0.smithi059.stdout:108: oid 15 version 15 is already newer than 13 2024-04-19T02:03:19.779 INFO:tasks.rados.rados.0.smithi059.stdout:108: oid 15 version 15 is already newer than 14 2024-04-19T02:03:19.779 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 15 v 15 (ObjNum 67 snap 3 seq_num 67) dirty exists 2024-04-19T02:03:19.779 INFO:tasks.rados.rados.0.smithi059.stdout:108: left oid 15 (ObjNum 67 snap 3 seq_num 67) 2024-04-19T02:03:19.780 INFO:tasks.rados.rados.0.smithi059.stdout:109: finishing write tid 1 to smithi05930959-41 2024-04-19T02:03:19.780 INFO:tasks.rados.rados.0.smithi059.stdout:109: finishing write tid 2 to smithi05930959-41 2024-04-19T02:03:19.780 INFO:tasks.rados.rados.0.smithi059.stdout:109: finishing write tid 3 to smithi05930959-41 2024-04-19T02:03:19.780 INFO:tasks.rados.rados.0.smithi059.stdout:109: finishing write tid 4 to smithi05930959-41 2024-04-19T02:03:19.780 INFO:tasks.rados.rados.0.smithi059.stdout:109: oid 41 updating version 0 to 25 2024-04-19T02:03:19.780 INFO:tasks.rados.rados.0.smithi059.stdout:109: oid 41 version 25 is already newer than 24 2024-04-19T02:03:19.780 INFO:tasks.rados.rados.0.smithi059.stdout:109: oid 41 version 25 is already newer than 23 2024-04-19T02:03:19.780 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 41 v 25 (ObjNum 68 snap 3 seq_num 68) dirty exists 2024-04-19T02:03:19.780 INFO:tasks.rados.rados.0.smithi059.stdout:109: left oid 41 (ObjNum 68 snap 3 seq_num 68) 2024-04-19T02:03:19.780 INFO:tasks.rados.rados.0.smithi059.stdout:105: done (4 left) 2024-04-19T02:03:19.780 INFO:tasks.rados.rados.0.smithi059.stdout:107: done (3 left) 2024-04-19T02:03:19.780 INFO:tasks.rados.rados.0.smithi059.stdout:108: done (2 left) 2024-04-19T02:03:19.780 INFO:tasks.rados.rados.0.smithi059.stdout:109: done (1 left) 2024-04-19T02:03:19.780 INFO:tasks.rados.rados.0.smithi059.stdout:112: delete oid 1 current snap is 3 2024-04-19T02:03:19.783 INFO:tasks.rados.rados.0.smithi059.stdout:112: done (1 left) 2024-04-19T02:03:19.783 INFO:tasks.rados.rados.0.smithi059.stdout:113: snap_create 2024-04-19T02:03:19.786 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-04-19T02:03:19.786 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T02:03:19.786 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 46 v 9 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-19T02:03:19.807 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-04-19T02:03:19.807 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T02:03:19.829 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-04-19T02:03:19.829 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T02:03:19.854 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-04-19T02:03:19.854 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T02:03:19.865 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:19.867+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:19.886 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:19.886 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:03:19.887 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-04-19T02:03:19.887 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-04-19T02:03:19.887 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:19.902 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-19T02:03:19.998 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:19.999+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:20.016 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:03:20.017 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-19T02:03:20.034 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-04-19T02:03:19.661222+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29655584,"num_objects":57,"num_object_clones":8,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":1,"num_read":289,"num_read_kb":13764,"num_write":372,"num_write_kb":33850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":342,"ondisk_log_size":342,"up":146,"acting":146,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1132462080,"kb_used":393264,"kb_used_data":60336,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1132068816,"statfs":{"total":1159641169920,"available":1159238467584,"internally_reserved":0,"allocated":61784064,"data_stored":66762006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":340374841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11223040,"num_objects":18,"num_object_clones":8,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":1,"num_read":63,"num_read_kb":9672,"num_write":76,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.030023"},"pg_stats":[{"pgid":"3.f","version":"34'13","reported_seq":63,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-19T02:03:18.698462+0000","last_change":"2024-04-19T02:03:18.705494+0000","last_active":"2024-04-19T02:03:18.698462+0000","last_peered":"2024-04-19T02:03:18.698462+0000","last_clean":"2024-04-19T02:03:18.698462+0000","last_became_active":"2024-04-19T02:03:11.728136+0000","last_became_peered":"2024-04-19T02:03:11.728136+0000","last_unstale":"2024-04-19T02:03:18.698462+0000","last_undegraded":"2024-04-19T02:03:18.698462+0000","last_fullsized":"2024-04-19T02:03:18.698462+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:16.534124+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.03300235,"stat_sum":{"num_bytes":1966080,"num_objects":3,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":2405,"num_write":10,"num_write_kb":1920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,2,3,5,8,0,7],"acting":[11,4,2,3,5,8,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"3.e","version":"34'16","reported_seq":63,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-19T02:03:18.711825+0000","last_change":"2024-04-19T02:03:18.711825+0000","last_active":"2024-04-19T02:03:18.711825+0000","last_peered":"2024-04-19T02:03:18.711825+0000","last_clean":"2024-04-19T02:03:18.711825+0000","last_became_active":"2024-04-19T02:03:11.730135+0000","last_became_peered":"2024-04-19T02:03:11.730135+0000","last_unstale":"2024-04-19T02:03:18.711825+0000","last_undegraded":"2024-04-19T02:03:18.711825+0000","last_fullsized":"2024-04-19T02:03:18.711825+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:39.160618+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.040218805000000003,"stat_sum":{"num_bytes":1523712,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":10,"num_write_kb":1680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5,11,0,2,3,9],"acting":[7,4,5,11,0,2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"3.d","version":"29'8","reported_seq":60,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-19T02:03:18.681602+0000","last_change":"2024-04-19T02:03:18.664128+0000","last_active":"2024-04-19T02:03:18.681602+0000","last_peered":"2024-04-19T02:03:18.681602+0000","last_clean":"2024-04-19T02:03:18.681602+0000","last_became_active":"2024-04-19T02:03:11.632682+0000","last_became_peered":"2024-04-19T02:03:11.632682+0000","last_unstale":"2024-04-19T02:03:18.681602+0000","last_undegraded":"2024-04-19T02:03:18.681602+0000","last_fullsized":"2024-04-19T02:03:18.681602+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:15.191112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001043932,"stat_sum":{"num_bytes":1261568,"num_objects":4,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":773,"num_write":8,"num_write_kb":1232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7,4,5,3,9,0],"acting":[8,11,7,4,5,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"3.c","version":"29'12","reported_seq":67,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-19T02:03:18.720446+0000","last_change":"2024-04-19T02:03:18.699161+0000","last_active":"2024-04-19T02:03:18.720446+0000","last_peered":"2024-04-19T02:03:18.720446+0000","last_clean":"2024-04-19T02:03:18.720446+0000","last_became_active":"2024-04-19T02:03:11.731568+0000","last_became_peered":"2024-04-19T02:03:11.731568+0000","last_unstale":"2024-04-19T02:03:18.720446+0000","last_undegraded":"2024-04-19T02:03:18.720446+0000","last_fullsized":"2024-04-19T02:03:18.720446+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:37.952067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.02725675,"stat_sum":{"num_bytes":2932736,"num_objects":4,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":4087,"num_write":12,"num_write_kb":2864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,11,7,4,2,0,5],"acting":[9,3,11,7,4,2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"3.b","version":"29'9","reported_seq":61,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-19T02:03:18.671029+0000","last_change":"2024-04-19T02:03:18.671029+0000","last_active":"2024-04-19T02:03:18.671029+0000","last_peered":"2024-04-19T02:03:18.671029+0000","last_clean":"2024-04-19T02:03:18.671029+0000","last_became_active":"2024-04-19T02:03:11.677316+0000","last_became_peered":"2024-04-19T02:03:11.677316+0000","last_unstale":"2024-04-19T02:03:18.671029+0000","last_undegraded":"2024-04-19T02:03:18.671029+0000","last_fullsized":"2024-04-19T02:03:18.671029+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:25.435484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027493119999999998,"stat_sum":{"num_bytes":2441216,"num_objects":3,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":245,"num_write":9,"num_write_kb":2384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8,6,4,10,1,7],"acting":[3,0,8,6,4,10,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"3.a","version":"34'22","reported_seq":70,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-19T02:03:18.707211+0000","last_change":"2024-04-19T02:03:18.709762+0000","last_active":"2024-04-19T02:03:18.707211+0000","last_peered":"2024-04-19T02:03:18.707211+0000","last_clean":"2024-04-19T02:03:18.707211+0000","last_became_active":"2024-04-19T02:03:11.727996+0000","last_became_peered":"2024-04-19T02:03:11.727996+0000","last_unstale":"2024-04-19T02:03:18.707211+0000","last_undegraded":"2024-04-19T02:03:18.707211+0000","last_fullsized":"2024-04-19T02:03:18.707211+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:32.910902+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.035679770999999999,"stat_sum":{"num_bytes":2392064,"num_objects":4,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":1685,"num_write":33,"num_write_kb":2336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,1,9,5,7,3,2],"acting":[11,8,1,9,5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"3.9","version":"34'19","reported_seq":65,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-19T02:03:18.698336+0000","last_change":"2024-04-19T02:03:18.701867+0000","last_active":"2024-04-19T02:03:18.698336+0000","last_peered":"2024-04-19T02:03:18.698336+0000","last_clean":"2024-04-19T02:03:18.698336+0000","last_became_active":"2024-04-19T02:03:11.729255+0000","last_became_peered":"2024-04-19T02:03:11.729255+0000","last_unstale":"2024-04-19T02:03:18.698336+0000","last_undegraded":"2024-04-19T02:03:18.698336+0000","last_fullsized":"2024-04-19T02:03:18.698336+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:37.151224+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.039622512999999998,"stat_sum":{"num_bytes":2260992,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":13,"num_write_kb":2944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,10,2,9,1,7,11],"acting":[4,3,10,2,9,1,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"3.8","version":"34'22","reported_seq":73,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-19T02:03:18.699299+0000","last_change":"2024-04-19T02:03:18.704377+0000","last_active":"2024-04-19T02:03:18.699299+0000","last_peered":"2024-04-19T02:03:18.699299+0000","last_clean":"2024-04-19T02:03:18.699299+0000","last_became_active":"2024-04-19T02:03:11.679384+0000","last_became_peered":"2024-04-19T02:03:11.679384+0000","last_unstale":"2024-04-19T02:03:18.699299+0000","last_undegraded":"2024-04-19T02:03:18.699299+0000","last_fullsized":"2024-04-19T02:03:18.699299+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:39.959498+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.042143987000000001,"stat_sum":{"num_bytes":2179072,"num_objects":6,"num_object_clones":2,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":17,"num_read_kb":871,"num_write":14,"num_write_kb":2128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7,11,5,9,6,10],"acting":[4,1,7,11,5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"1.0","version":"21'137","reported_seq":213,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-19T02:03:18.662475+0000","last_change":"2024-04-19T02:02:59.974166+0000","last_active":"2024-04-19T02:03:18.662475+0000","last_peered":"2024-04-19T02:03:18.662475+0000","last_clean":"2024-04-19T02:03:18.662475+0000","last_became_active":"2024-04-19T02:02:59.972005+0000","last_became_peered":"2024-04-19T02:02:59.972005+0000","last_unstale":"2024-04-19T02:03:18.662475+0000","last_undegraded":"2024-04-19T02:03:18.662475+0000","last_fullsized":"2024-04-19T02:03:18.662475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:02:58.486225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:02:58.486225+0000","last_clean_scrub_stamp":"2024-04-19T02:02:58.486225+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:20.275707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":34,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-19T02:03:18.671603+0000","last_change":"2024-04-19T02:03:03.026806+0000","last_active":"2024-04-19T02:03:18.671603+0000","last_peered":"2024-04-19T02:03:18.671603+0000","last_clean":"2024-04-19T02:03:18.671603+0000","last_became_active":"2024-04-19T02:03:03.026063+0000","last_became_peered":"2024-04-19T02:03:03.026063+0000","last_unstale":"2024-04-19T02:03:18.671603+0000","last_undegraded":"2024-04-19T02:03:18.671603+0000","last_fullsized":"2024-04-19T02:03:18.671603+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:19.673016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"34'44","reported_seq":98,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-19T02:03:18.723475+0000","last_change":"2024-04-19T02:03:18.724402+0000","last_active":"2024-04-19T02:03:18.723475+0000","last_peered":"2024-04-19T02:03:18.723475+0000","last_clean":"2024-04-19T02:03:18.723475+0000","last_became_active":"2024-04-19T02:03:11.630475+0000","last_became_peered":"2024-04-19T02:03:11.630475+0000","last_unstale":"2024-04-19T02:03:18.723475+0000","last_undegraded":"2024-04-19T02:03:18.723475+0000","last_fullsized":"2024-04-19T02:03:18.723475+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:16.974693+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.055694894000000002,"stat_sum":{"num_bytes":3801088,"num_objects":7,"num_object_clones":1,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":1548,"num_write":47,"num_write_kb":5312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10,4,8,1,11,9],"acting":[3,5,10,4,8,1,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":34,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-19T02:03:18.662377+0000","last_change":"2024-04-19T02:03:02.561076+0000","last_active":"2024-04-19T02:03:18.662377+0000","last_peered":"2024-04-19T02:03:18.662377+0000","last_clean":"2024-04-19T02:03:18.662377+0000","last_became_active":"2024-04-19T02:03:02.560210+0000","last_became_peered":"2024-04-19T02:03:02.560210+0000","last_unstale":"2024-04-19T02:03:18.662377+0000","last_undegraded":"2024-04-19T02:03:18.662377+0000","last_fullsized":"2024-04-19T02:03:18.662377+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:16.900004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"29'6","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-19T02:03:14.659361+0000","last_change":"2024-04-19T02:03:11.634283+0000","last_active":"2024-04-19T02:03:14.659361+0000","last_peered":"2024-04-19T02:03:14.659361+0000","last_clean":"2024-04-19T02:03:14.659361+0000","last_became_active":"2024-04-19T02:03:11.633394+0000","last_became_peered":"2024-04-19T02:03:11.633394+0000","last_unstale":"2024-04-19T02:03:14.659361+0000","last_undegraded":"2024-04-19T02:03:14.659361+0000","last_fullsized":"2024-04-19T02:03:14.659361+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:32.127833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1589248,"num_objects":3,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":1348,"num_write":6,"num_write_kb":1552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,4,5,10,7,9,1],"acting":[0,11,4,5,10,7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":29,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-19T02:03:18.668199+0000","last_change":"2024-04-19T02:03:09.594947+0000","last_active":"2024-04-19T02:03:18.668199+0000","last_peered":"2024-04-19T02:03:18.668199+0000","last_clean":"2024-04-19T02:03:18.668199+0000","last_became_active":"2024-04-19T02:03:09.594072+0000","last_became_peered":"2024-04-19T02:03:09.594072+0000","last_unstale":"2024-04-19T02:03:18.668199+0000","last_undegraded":"2024-04-19T02:03:18.668199+0000","last_fullsized":"2024-04-19T02:03:18.668199+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:29.461650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":37,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-19T02:03:17.656346+0000","last_change":"2024-04-19T02:03:16.654956+0000","last_active":"2024-04-19T02:03:17.656346+0000","last_peered":"2024-04-19T02:03:17.656346+0000","last_clean":"2024-04-19T02:03:17.656346+0000","last_became_active":"2024-04-19T02:03:11.678078+0000","last_became_peered":"2024-04-19T02:03:11.678078+0000","last_unstale":"2024-04-19T02:03:17.656346+0000","last_undegraded":"2024-04-19T02:03:17.656346+0000","last_fullsized":"2024-04-19T02:03:17.656346+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:27.063638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010554889999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,9,1,6,4,0,10],"acting":[5,7,9,1,6,4,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":34,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-19T02:03:18.662735+0000","last_change":"2024-04-19T02:03:02.560133+0000","last_active":"2024-04-19T02:03:18.662735+0000","last_peered":"2024-04-19T02:03:18.662735+0000","last_clean":"2024-04-19T02:03:18.662735+0000","last_became_active":"2024-04-19T02:03:02.559377+0000","last_became_peered":"2024-04-19T02:03:02.559377+0000","last_unstale":"2024-04-19T02:03:18.662735+0000","last_undegraded":"2024-04-19T02:03:18.662735+0000","last_fullsized":"2024-04-19T02:03:18.662735+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:30.728400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"29'4","reported_seq":43,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-19T02:03:17.646508+0000","last_change":"2024-04-19T02:03:16.665112+0000","last_active":"2024-04-19T02:03:17.646508+0000","last_peered":"2024-04-19T02:03:17.646508+0000","last_clean":"2024-04-19T02:03:17.646508+0000","last_became_active":"2024-04-19T02:03:11.677099+0000","last_became_peered":"2024-04-19T02:03:11.677099+0000","last_unstale":"2024-04-19T02:03:17.646508+0000","last_undegraded":"2024-04-19T02:03:17.646508+0000","last_fullsized":"2024-04-19T02:03:17.646508+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:15.554073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001361623,"stat_sum":{"num_bytes":802816,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":4,"num_write_kb":784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,0,3,4,7,9,6],"acting":[10,8,0,3,4,7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"3.5","version":"30'10","reported_seq":49,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-19T02:03:17.646566+0000","last_change":"2024-04-19T02:03:16.665325+0000","last_active":"2024-04-19T02:03:17.646566+0000","last_peered":"2024-04-19T02:03:17.646566+0000","last_clean":"2024-04-19T02:03:17.646566+0000","last_became_active":"2024-04-19T02:03:11.728949+0000","last_became_peered":"2024-04-19T02:03:11.728949+0000","last_unstale":"2024-04-19T02:03:17.646566+0000","last_undegraded":"2024-04-19T02:03:17.646566+0000","last_fullsized":"2024-04-19T02:03:17.646566+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:13.453042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012563209999999999,"stat_sum":{"num_bytes":1327104,"num_objects":4,"num_object_clones":1,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":8,"num_write_kb":1296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2,9,8,1,6,5],"acting":[10,3,2,9,8,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-19T02:03:17.654725+0000","last_change":"2024-04-19T02:03:02.597582+0000","last_active":"2024-04-19T02:03:17.654725+0000","last_peered":"2024-04-19T02:03:17.654725+0000","last_clean":"2024-04-19T02:03:17.654725+0000","last_became_active":"2024-04-19T02:03:02.597118+0000","last_became_peered":"2024-04-19T02:03:02.597118+0000","last_unstale":"2024-04-19T02:03:17.654725+0000","last_undegraded":"2024-04-19T02:03:17.654725+0000","last_fullsized":"2024-04-19T02:03:17.654725+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:13.683143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"33'14","reported_seq":59,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-19T02:03:17.668678+0000","last_change":"2024-04-19T02:03:16.666695+0000","last_active":"2024-04-19T02:03:17.668678+0000","last_peered":"2024-04-19T02:03:17.668678+0000","last_clean":"2024-04-19T02:03:17.668678+0000","last_became_active":"2024-04-19T02:03:11.730165+0000","last_became_peered":"2024-04-19T02:03:11.730165+0000","last_unstale":"2024-04-19T02:03:17.668678+0000","last_undegraded":"2024-04-19T02:03:17.668678+0000","last_fullsized":"2024-04-19T02:03:17.668678+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:36.897572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014160240000000001,"stat_sum":{"num_bytes":3293184,"num_objects":7,"num_object_clones":2,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":16,"num_read_kb":678,"num_write":12,"num_write_kb":3216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,10,8,7,2,11,0],"acting":[1,5,10,8,7,2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":34,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-19T02:03:18.672207+0000","last_change":"2024-04-19T02:03:02.595929+0000","last_active":"2024-04-19T02:03:18.672207+0000","last_peered":"2024-04-19T02:03:18.672207+0000","last_clean":"2024-04-19T02:03:18.672207+0000","last_became_active":"2024-04-19T02:03:02.595232+0000","last_became_peered":"2024-04-19T02:03:02.595232+0000","last_unstale":"2024-04-19T02:03:18.672207+0000","last_undegraded":"2024-04-19T02:03:18.672207+0000","last_fullsized":"2024-04-19T02:03:18.672207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:04.077370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"34'4","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-19T02:03:18.701415+0000","last_change":"2024-04-19T02:03:18.672912+0000","last_active":"2024-04-19T02:03:18.701415+0000","last_peered":"2024-04-19T02:03:18.701415+0000","last_clean":"2024-04-19T02:03:18.701415+0000","last_became_active":"2024-04-19T02:03:11.675056+0000","last_became_peered":"2024-04-19T02:03:11.675056+0000","last_unstale":"2024-04-19T02:03:18.701415+0000","last_undegraded":"2024-04-19T02:03:18.701415+0000","last_fullsized":"2024-04-19T02:03:18.701415+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:37.793769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043986340000000002,"stat_sum":{"num_bytes":1114112,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":3,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,9,0,7,5,11,1],"acting":[3,6,9,0,7,5,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":32,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-19T02:03:17.654529+0000","last_change":"2024-04-19T02:03:02.988302+0000","last_active":"2024-04-19T02:03:17.654529+0000","last_peered":"2024-04-19T02:03:17.654529+0000","last_clean":"2024-04-19T02:03:17.654529+0000","last_became_active":"2024-04-19T02:03:02.988073+0000","last_became_peered":"2024-04-19T02:03:02.988073+0000","last_unstale":"2024-04-19T02:03:17.654529+0000","last_undegraded":"2024-04-19T02:03:17.654529+0000","last_fullsized":"2024-04-19T02:03:17.654529+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:28.455271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"29'2","reported_seq":27,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-19T02:03:14.659801+0000","last_change":"2024-04-19T02:03:11.675325+0000","last_active":"2024-04-19T02:03:14.659801+0000","last_peered":"2024-04-19T02:03:14.659801+0000","last_clean":"2024-04-19T02:03:14.659801+0000","last_became_active":"2024-04-19T02:03:11.673940+0000","last_became_peered":"2024-04-19T02:03:11.673940+0000","last_unstale":"2024-04-19T02:03:14.659801+0000","last_undegraded":"2024-04-19T02:03:14.659801+0000","last_fullsized":"2024-04-19T02:03:14.659801+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:17.381273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180224,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9,8,4,5,6,10],"acting":[0,1,9,8,4,5,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":34,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-19T02:03:18.663125+0000","last_change":"2024-04-19T02:03:02.570839+0000","last_active":"2024-04-19T02:03:18.663125+0000","last_peered":"2024-04-19T02:03:18.663125+0000","last_clean":"2024-04-19T02:03:18.663125+0000","last_became_active":"2024-04-19T02:03:02.570116+0000","last_became_peered":"2024-04-19T02:03:02.570116+0000","last_unstale":"2024-04-19T02:03:18.663125+0000","last_undegraded":"2024-04-19T02:03:18.663125+0000","last_fullsized":"2024-04-19T02:03:18.663125+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:23.134425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":29065216,"num_objects":55,"num_object_clones":8,"num_object_copies":440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":1,"num_read":163,"num_read_kb":13655,"num_write":191,"num_write_kb":30912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57835520,"data_stored":65810432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":128,"acting":128,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":19,"seq":81604378631,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30140,"kb_used_data":3120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341700,"statfs":{"total":96636764160,"available":96605900800,"internally_reserved":0,"allocated":3194880,"data_stored":3195665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":19,"seq":81604378630,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31088,"kb_used_data":4064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340752,"statfs":{"total":96636764160,"available":96604930048,"internally_reserved":0,"allocated":4161536,"data_stored":4129553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411335,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33248,"kb_used_data":6228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338592,"statfs":{"total":96636764160,"available":96602718208,"internally_reserved":0,"allocated":6377472,"data_stored":6706793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411336,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32584,"kb_used_data":5564,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339256,"statfs":{"total":96636764160,"available":96603398144,"internally_reserved":0,"allocated":5697536,"data_stored":6219369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411336,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32104,"kb_used_data":5076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339736,"statfs":{"total":96636764160,"available":96603889664,"internally_reserved":0,"allocated":5197824,"data_stored":5294217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411336,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33116,"kb_used_data":6092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338724,"statfs":{"total":96636764160,"available":96602853376,"internally_reserved":0,"allocated":6238208,"data_stored":6674025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":18,"seq":77309411334,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29148,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342692,"statfs":{"total":96636764160,"available":96606916608,"internally_reserved":0,"allocated":2174976,"data_stored":2741616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411335,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32820,"kb_used_data":5460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339020,"statfs":{"total":96636764160,"available":96603156480,"internally_reserved":0,"allocated":5591040,"data_stored":6415977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":18,"seq":77309411335,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32700,"kb_used_data":5676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339140,"statfs":{"total":96636764160,"available":96603279360,"internally_reserved":0,"allocated":5812224,"data_stored":5912169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":17,"seq":73014444039,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31920,"kb_used_data":4900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339920,"statfs":{"total":96636764160,"available":96604078080,"internally_reserved":0,"allocated":5017600,"data_stored":5387439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":17,"seq":73014444039,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37828,"kb_used_data":6648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334012,"statfs":{"total":96636764160,"available":96598028288,"internally_reserved":0,"allocated":6807552,"data_stored":7919504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":17,"seq":73014444039,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36568,"kb_used_data":5384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335272,"statfs":{"total":96636764160,"available":96599318528,"internally_reserved":0,"allocated":5513216,"data_stored":6165679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3854336,"data_stored":3985408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5304320,"data_stored":6549504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3682304,"data_stored":4333568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5455872,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5234688,"data_stored":5935104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":5931008,"data_stored":7221248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2011136,"data_stored":2625536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5881856,"data_stored":6496256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4308992,"data_stored":4587520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":5341184,"data_stored":6041600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4808704,"data_stored":5771264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":6021120,"data_stored":6529024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T02:03:20.035 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pgp_num 16 2024-04-19T02:03:20.065 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:20.065 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-04-19T02:03:20.065 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-04-19T02:03:20.065 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-04-19T02:03:20.065 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.065 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "pg_update_log_missing(3.es5 epoch 32/26 rep_tid 18 entries 32'12 (0'0) error 3:79fe7502:::smithi05930959-12:head by client.4509.0:240 0.000000 -2 [r=-2+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 30'10)", 2024-04-19T02:03:20.065 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.651404+0000", 2024-04-19T02:03:20.065 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 3.416324076, 2024-04-19T02:03:20.065 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001738641, 2024-04-19T02:03:20.066 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.066 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.066 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:20.066 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.066 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.066 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.066 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.651404+0000", 2024-04-19T02:03:20.066 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.066 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.066 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.066 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.066 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.651404+0000", 2024-04-19T02:03:20.066 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.066 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.067 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.067 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.067 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.651405+0000", 2024-04-19T02:03:20.067 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.5000000000000002e-07 2024-04-19T02:03:20.067 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.067 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.067 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.067 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.651428+0000", 2024-04-19T02:03:20.067 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3523000000000002e-05 2024-04-19T02:03:20.067 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.067 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.068 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.068 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.651496+0000", 2024-04-19T02:03:20.068 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8108000000000006e-05 2024-04-19T02:03:20.068 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.068 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.068 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.068 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.651594+0000", 2024-04-19T02:03:20.068 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.8016999999999993e-05 2024-04-19T02:03:20.068 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.069 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.069 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.069 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.651832+0000", 2024-04-19T02:03:20.069 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00023731300000000001 2024-04-19T02:03:20.069 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.069 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.069 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.069 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.653143+0000", 2024-04-19T02:03:20.069 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0013110299999999999 2024-04-19T02:03:20.069 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.069 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.069 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.069 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.069 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.070 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s5 35/26 ECSubWrite(tid=29, reqid=osd.1.0:29, at_version=35'18, trim_to=0'0, roll_forward_to=34'16))", 2024-04-19T02:03:20.070 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.667338+0000", 2024-04-19T02:03:20.070 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.40038991400000001, 2024-04-19T02:03:20.070 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002196726, 2024-04-19T02:03:20.070 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.070 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.070 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:20.070 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.070 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.070 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.070 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.667338+0000", 2024-04-19T02:03:20.070 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.070 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.070 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.070 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.071 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.667338+0000", 2024-04-19T02:03:20.071 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.071 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.071 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.071 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.071 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.667338+0000", 2024-04-19T02:03:20.071 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2700000000000001e-07 2024-04-19T02:03:20.071 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.071 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.071 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.071 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.667362+0000", 2024-04-19T02:03:20.071 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3343e-05 2024-04-19T02:03:20.072 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.072 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.072 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.072 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.667375+0000", 2024-04-19T02:03:20.072 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3750999999999999e-05 2024-04-19T02:03:20.072 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.072 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.072 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.072 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.667386+0000", 2024-04-19T02:03:20.072 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0159e-05 2024-04-19T02:03:20.072 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.072 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.072 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.072 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.667497+0000", 2024-04-19T02:03:20.072 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000111229 2024-04-19T02:03:20.073 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.073 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.073 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:20.073 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.667507+0000", 2024-04-19T02:03:20.073 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0387000000000001e-05 2024-04-19T02:03:20.073 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.073 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.073 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:20.073 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.669488+0000", 2024-04-19T02:03:20.073 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0019807650000000002 2024-04-19T02:03:20.073 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.073 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.073 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.073 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.669535+0000", 2024-04-19T02:03:20.073 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.6864999999999999e-05 2024-04-19T02:03:20.074 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.074 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.074 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.074 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.074 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.074 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es5 35/26 ECSubWrite(tid=23, reqid=osd.7.0:23, at_version=35'18, trim_to=0'0, roll_forward_to=34'16))", 2024-04-19T02:03:20.074 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.674564+0000", 2024-04-19T02:03:20.074 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.39316407399999997, 2024-04-19T02:03:20.074 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0023738539999999999, 2024-04-19T02:03:20.074 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.074 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.074 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:20.074 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.074 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.075 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.075 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.674564+0000", 2024-04-19T02:03:20.075 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.075 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.075 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.075 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.075 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.674564+0000", 2024-04-19T02:03:20.075 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.075 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.075 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.075 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.075 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.674564+0000", 2024-04-19T02:03:20.075 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.3599999999999999e-07 2024-04-19T02:03:20.075 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.075 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.076 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.076 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.674595+0000", 2024-04-19T02:03:20.076 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0273999999999998e-05 2024-04-19T02:03:20.076 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.076 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.076 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.076 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.674614+0000", 2024-04-19T02:03:20.076 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9147000000000002e-05 2024-04-19T02:03:20.076 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.076 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.076 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.076 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.674625+0000", 2024-04-19T02:03:20.076 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1056999999999999e-05 2024-04-19T02:03:20.076 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.077 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.077 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.077 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.674723+0000", 2024-04-19T02:03:20.077 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.8010999999999994e-05 2024-04-19T02:03:20.077 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.077 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.077 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:20.077 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.674733+0000", 2024-04-19T02:03:20.077 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.045e-05 2024-04-19T02:03:20.077 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.077 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.077 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:20.077 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.676896+0000", 2024-04-19T02:03:20.077 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002162481 2024-04-19T02:03:20.077 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.078 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.078 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.078 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.676938+0000", 2024-04-19T02:03:20.078 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1998e-05 2024-04-19T02:03:20.078 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.078 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.078 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.078 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.078 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.078 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s5 35/26 ECSubWrite(tid=30, reqid=osd.1.0:29, at_version=0'0, trim_to=0'0, roll_forward_to=35'18))", 2024-04-19T02:03:20.078 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.680759+0000", 2024-04-19T02:03:20.078 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.38696857299999998, 2024-04-19T02:03:20.078 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0036058090000000002, 2024-04-19T02:03:20.078 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.079 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.079 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:20.079 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.079 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.079 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.079 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680759+0000", 2024-04-19T02:03:20.079 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.079 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.079 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.079 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.079 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680759+0000", 2024-04-19T02:03:20.079 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.079 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.079 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.079 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.080 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680761+0000", 2024-04-19T02:03:20.080 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2929999999999999e-06 2024-04-19T02:03:20.080 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.080 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.080 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.080 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680775+0000", 2024-04-19T02:03:20.080 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4284000000000001e-05 2024-04-19T02:03:20.080 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.080 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.080 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.080 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680793+0000", 2024-04-19T02:03:20.080 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8176999999999999e-05 2024-04-19T02:03:20.080 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.080 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.080 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.081 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680804+0000", 2024-04-19T02:03:20.081 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0508e-05 2024-04-19T02:03:20.081 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.081 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.081 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.081 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680903+0000", 2024-04-19T02:03:20.081 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9370999999999994e-05 2024-04-19T02:03:20.081 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.081 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.081 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:20.081 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680914+0000", 2024-04-19T02:03:20.081 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1255e-05 2024-04-19T02:03:20.081 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.081 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.081 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:20.082 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.684325+0000", 2024-04-19T02:03:20.082 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.003410471 2024-04-19T02:03:20.082 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.082 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.082 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.082 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.684365+0000", 2024-04-19T02:03:20.082 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0450000000000001e-05 2024-04-19T02:03:20.082 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.082 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.082 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.082 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.082 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.082 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs2 35/26 ECSubWrite(tid=54, reqid=client.4509.0:271, at_version=35'14, trim_to=0'0, roll_forward_to=34'13))", 2024-04-19T02:03:20.082 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.684827+0000", 2024-04-19T02:03:20.083 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.38290077099999997, 2024-04-19T02:03:20.083 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.003378519, 2024-04-19T02:03:20.083 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.083 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.083 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:20.083 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.083 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.083 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.083 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.684827+0000", 2024-04-19T02:03:20.083 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.083 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.083 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.083 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.083 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.684827+0000", 2024-04-19T02:03:20.083 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.084 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.084 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.084 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.084 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.684827+0000", 2024-04-19T02:03:20.084 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7000000000000001e-07 2024-04-19T02:03:20.084 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.084 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.084 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.084 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.685370+0000", 2024-04-19T02:03:20.084 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00054286000000000004 2024-04-19T02:03:20.084 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.084 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.084 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.084 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.685390+0000", 2024-04-19T02:03:20.084 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9346000000000001e-05 2024-04-19T02:03:20.085 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.085 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.085 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.085 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.685400+0000", 2024-04-19T02:03:20.085 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0079e-05 2024-04-19T02:03:20.085 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.085 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.085 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.085 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.685500+0000", 2024-04-19T02:03:20.085 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000100074 2024-04-19T02:03:20.085 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.085 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.085 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:20.085 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.685511+0000", 2024-04-19T02:03:20.085 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1345999999999999e-05 2024-04-19T02:03:20.086 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.086 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.086 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:20.086 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.688159+0000", 2024-04-19T02:03:20.086 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0026475159999999999 2024-04-19T02:03:20.086 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.086 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.086 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.086 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.688206+0000", 2024-04-19T02:03:20.086 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.7027999999999998e-05 2024-04-19T02:03:20.086 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.086 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.086 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.086 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.086 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.087 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs2 35/26 ECSubWrite(tid=55, reqid=client.4509.0:272, at_version=35'15, trim_to=0'0, roll_forward_to=34'13))", 2024-04-19T02:03:20.087 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.689508+0000", 2024-04-19T02:03:20.087 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.37821969, 2024-04-19T02:03:20.087 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0029708820000000002, 2024-04-19T02:03:20.087 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.087 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.087 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:20.087 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.087 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.087 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.087 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.689508+0000", 2024-04-19T02:03:20.087 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.087 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.087 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.087 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.088 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.689508+0000", 2024-04-19T02:03:20.088 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.088 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.088 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.088 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.088 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.689510+0000", 2024-04-19T02:03:20.088 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2610000000000001e-06 2024-04-19T02:03:20.088 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.088 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.088 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.088 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.689634+0000", 2024-04-19T02:03:20.088 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000124635 2024-04-19T02:03:20.088 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.088 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.088 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.089 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.689654+0000", 2024-04-19T02:03:20.089 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0024999999999999e-05 2024-04-19T02:03:20.089 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.089 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.089 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.089 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.689664+0000", 2024-04-19T02:03:20.089 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0061000000000001e-05 2024-04-19T02:03:20.089 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.089 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.089 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.089 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.689751+0000", 2024-04-19T02:03:20.089 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.6637000000000004e-05 2024-04-19T02:03:20.089 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.089 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.089 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:20.090 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.689761+0000", 2024-04-19T02:03:20.090 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0261000000000001e-05 2024-04-19T02:03:20.090 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.090 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.090 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:20.090 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.692434+0000", 2024-04-19T02:03:20.090 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0026723490000000001 2024-04-19T02:03:20.090 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.090 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.090 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.090 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.692479+0000", 2024-04-19T02:03:20.090 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5652999999999998e-05 2024-04-19T02:03:20.090 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.090 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.090 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.091 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.091 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.091 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs2 35/26 ECSubWrite(tid=56, reqid=client.4509.0:273, at_version=35'16, trim_to=0'0, roll_forward_to=34'13))", 2024-04-19T02:03:20.091 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.695910+0000", 2024-04-19T02:03:20.091 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.37181784299999998, 2024-04-19T02:03:20.091 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00083400100000000001, 2024-04-19T02:03:20.091 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.091 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.091 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:20.091 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.091 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.091 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.091 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.695910+0000", 2024-04-19T02:03:20.091 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.092 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.092 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.092 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.092 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.695910+0000", 2024-04-19T02:03:20.092 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.092 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.092 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.092 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.092 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.695911+0000", 2024-04-19T02:03:20.092 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0839999999999999e-06 2024-04-19T02:03:20.092 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.092 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.092 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.092 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.695933+0000", 2024-04-19T02:03:20.093 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1596000000000001e-05 2024-04-19T02:03:20.093 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.093 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.093 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.093 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.695957+0000", 2024-04-19T02:03:20.093 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3847999999999999e-05 2024-04-19T02:03:20.093 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.093 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.093 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.093 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.695969+0000", 2024-04-19T02:03:20.093 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1975e-05 2024-04-19T02:03:20.093 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.093 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.093 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.093 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.696067+0000", 2024-04-19T02:03:20.094 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.8815999999999997e-05 2024-04-19T02:03:20.094 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.094 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.094 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:20.094 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.696083+0000", 2024-04-19T02:03:20.094 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5753999999999999e-05 2024-04-19T02:03:20.094 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.094 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.094 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:20.094 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.696699+0000", 2024-04-19T02:03:20.094 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00061576100000000004 2024-04-19T02:03:20.094 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.094 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.094 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.094 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.696744+0000", 2024-04-19T02:03:20.095 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5167000000000003e-05 2024-04-19T02:03:20.095 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.095 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.095 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.095 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.095 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.095 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es5 35/26 ECSubWrite(tid=24, reqid=osd.7.0:23, at_version=0'0, trim_to=0'0, roll_forward_to=35'18))", 2024-04-19T02:03:20.095 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.696711+0000", 2024-04-19T02:03:20.095 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.371016809, 2024-04-19T02:03:20.095 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002790196, 2024-04-19T02:03:20.095 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.095 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.095 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:20.096 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.096 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.096 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.096 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.696711+0000", 2024-04-19T02:03:20.096 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.096 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.096 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.096 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.096 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.696711+0000", 2024-04-19T02:03:20.096 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.096 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.096 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.096 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.096 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.696712+0000", 2024-04-19T02:03:20.096 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2799999999999995e-07 2024-04-19T02:03:20.096 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.097 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.097 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.097 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.696725+0000", 2024-04-19T02:03:20.097 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3064000000000001e-05 2024-04-19T02:03:20.097 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.097 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.097 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.097 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.696738+0000", 2024-04-19T02:03:20.097 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3199e-05 2024-04-19T02:03:20.097 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.097 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.097 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.097 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.696748+0000", 2024-04-19T02:03:20.097 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.7850000000000003e-06 2024-04-19T02:03:20.097 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.097 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.098 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.098 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.696828+0000", 2024-04-19T02:03:20.098 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0117999999999994e-05 2024-04-19T02:03:20.098 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.098 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.098 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:20.098 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.696837+0000", 2024-04-19T02:03:20.098 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0569999999999999e-06 2024-04-19T02:03:20.098 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.098 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.098 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:20.098 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.699452+0000", 2024-04-19T02:03:20.098 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0026147589999999999 2024-04-19T02:03:20.098 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.099 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.099 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.099 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.699501+0000", 2024-04-19T02:03:20.099 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9385999999999998e-05 2024-04-19T02:03:20.099 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.099 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.099 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.099 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.099 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.099 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs2 35/26 ECSubWrite(tid=57, reqid=client.4509.0:273, at_version=0'0, trim_to=0'0, roll_forward_to=35'16))", 2024-04-19T02:03:20.099 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.702768+0000", 2024-04-19T02:03:20.099 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.364960167, 2024-04-19T02:03:20.099 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00083436000000000003, 2024-04-19T02:03:20.099 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.099 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.100 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:20.100 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.100 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.100 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.100 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.702768+0000", 2024-04-19T02:03:20.100 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.100 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.100 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.100 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.100 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.702768+0000", 2024-04-19T02:03:20.100 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.100 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.100 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.100 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.100 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.702769+0000", 2024-04-19T02:03:20.101 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0599999999999999e-07 2024-04-19T02:03:20.101 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.101 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.101 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.101 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.702782+0000", 2024-04-19T02:03:20.101 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2815e-05 2024-04-19T02:03:20.101 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.101 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.101 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.101 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.702799+0000", 2024-04-19T02:03:20.101 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7014999999999999e-05 2024-04-19T02:03:20.101 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.101 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.101 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.101 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.702810+0000", 2024-04-19T02:03:20.101 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1909999999999999e-05 2024-04-19T02:03:20.102 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.102 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.102 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.102 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.702876+0000", 2024-04-19T02:03:20.102 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.5712999999999999e-05 2024-04-19T02:03:20.102 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.102 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.102 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:20.102 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.702886+0000", 2024-04-19T02:03:20.102 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.4700000000000008e-06 2024-04-19T02:03:20.102 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.102 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.102 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:20.102 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.703556+0000", 2024-04-19T02:03:20.103 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00067058500000000002 2024-04-19T02:03:20.103 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.103 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.103 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.103 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.703602+0000", 2024-04-19T02:03:20.103 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5945999999999997e-05 2024-04-19T02:03:20.103 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.103 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.103 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.103 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.103 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.103 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.5s2 35/26 ECSubWrite(tid=31, reqid=client.4509.0:281, at_version=35'13, trim_to=0'0, roll_forward_to=34'12))", 2024-04-19T02:03:20.103 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.740829+0000", 2024-04-19T02:03:20.103 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.32689906299999999, 2024-04-19T02:03:20.103 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001782386, 2024-04-19T02:03:20.104 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.104 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.104 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:20.104 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.104 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.104 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.104 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.740829+0000", 2024-04-19T02:03:20.104 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.104 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.104 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.104 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.104 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.740829+0000", 2024-04-19T02:03:20.104 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.104 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.104 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.105 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.105 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.740829+0000", 2024-04-19T02:03:20.105 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.7899999999999999e-07 2024-04-19T02:03:20.105 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.105 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.105 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.105 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.741403+0000", 2024-04-19T02:03:20.105 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000573238 2024-04-19T02:03:20.105 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.105 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.105 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.105 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.741421+0000", 2024-04-19T02:03:20.105 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8953e-05 2024-04-19T02:03:20.105 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.106 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.106 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.106 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.741433+0000", 2024-04-19T02:03:20.106 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1683e-05 2024-04-19T02:03:20.106 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.106 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.106 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.106 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.741505+0000", 2024-04-19T02:03:20.106 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.2028999999999998e-05 2024-04-19T02:03:20.106 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.106 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.106 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:20.106 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.741516+0000", 2024-04-19T02:03:20.106 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0864e-05 2024-04-19T02:03:20.106 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.107 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.107 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:20.107 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.742568+0000", 2024-04-19T02:03:20.107 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0010523869999999999 2024-04-19T02:03:20.107 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.107 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.107 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.107 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.742611+0000", 2024-04-19T02:03:20.107 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2852999999999998e-05 2024-04-19T02:03:20.107 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.107 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.107 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.107 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.107 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.107 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.5s2 35/26 ECSubWrite(tid=32, reqid=client.4509.0:282, at_version=35'14, trim_to=0'0, roll_forward_to=34'12))", 2024-04-19T02:03:20.108 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.745030+0000", 2024-04-19T02:03:20.108 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.32269841999999999, 2024-04-19T02:03:20.108 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001431405, 2024-04-19T02:03:20.108 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.108 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.108 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:20.108 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.108 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.108 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.108 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.745030+0000", 2024-04-19T02:03:20.108 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.108 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.108 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.108 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.109 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.745030+0000", 2024-04-19T02:03:20.109 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.109 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.109 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.109 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.109 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.745031+0000", 2024-04-19T02:03:20.109 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3659999999999999e-06 2024-04-19T02:03:20.109 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.109 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.109 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.109 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.745127+0000", 2024-04-19T02:03:20.109 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.5964999999999993e-05 2024-04-19T02:03:20.109 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.109 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.109 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.110 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.745151+0000", 2024-04-19T02:03:20.110 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3927999999999999e-05 2024-04-19T02:03:20.110 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.110 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.110 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.110 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.745165+0000", 2024-04-19T02:03:20.110 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.438e-05 2024-04-19T02:03:20.110 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.110 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.110 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.110 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.745272+0000", 2024-04-19T02:03:20.110 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010671 2024-04-19T02:03:20.110 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.110 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.111 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:20.111 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.745282+0000", 2024-04-19T02:03:20.111 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.6749999999999997e-06 2024-04-19T02:03:20.111 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.111 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.111 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:20.111 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.746397+0000", 2024-04-19T02:03:20.111 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0011156790000000001 2024-04-19T02:03:20.111 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.111 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.111 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.111 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.746461+0000", 2024-04-19T02:03:20.111 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.3701999999999994e-05 2024-04-19T02:03:20.111 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.111 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.112 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.112 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.112 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.112 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.5s2 35/26 ECSubWrite(tid=33, reqid=client.4509.0:283, at_version=35'15, trim_to=0'0, roll_forward_to=34'12))", 2024-04-19T02:03:20.112 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.747895+0000", 2024-04-19T02:03:20.112 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.31983266199999999, 2024-04-19T02:03:20.112 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000862659, 2024-04-19T02:03:20.112 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.112 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.112 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:20.112 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.112 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.112 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.112 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.747895+0000", 2024-04-19T02:03:20.112 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.113 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.113 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.113 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.113 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.747895+0000", 2024-04-19T02:03:20.113 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.113 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.113 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.113 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.113 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.747897+0000", 2024-04-19T02:03:20.113 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.316e-06 2024-04-19T02:03:20.113 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.113 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.113 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.113 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.747922+0000", 2024-04-19T02:03:20.113 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5814000000000001e-05 2024-04-19T02:03:20.114 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.114 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.114 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.114 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.747941+0000", 2024-04-19T02:03:20.114 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8077999999999999e-05 2024-04-19T02:03:20.114 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.114 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.114 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.114 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.747952+0000", 2024-04-19T02:03:20.114 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1718e-05 2024-04-19T02:03:20.114 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.114 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.114 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.114 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.748053+0000", 2024-04-19T02:03:20.114 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000100342 2024-04-19T02:03:20.115 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.115 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.115 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:20.115 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.748063+0000", 2024-04-19T02:03:20.115 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0054999999999999e-05 2024-04-19T02:03:20.115 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.115 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.115 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:20.115 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.748730+0000", 2024-04-19T02:03:20.115 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00066752100000000002 2024-04-19T02:03:20.115 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.115 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.115 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.115 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.748758+0000", 2024-04-19T02:03:20.115 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7815000000000001e-05 2024-04-19T02:03:20.116 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.116 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.116 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.116 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.116 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.116 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.5s2 35/26 ECSubWrite(tid=34, reqid=client.4509.0:283, at_version=0'0, trim_to=0'0, roll_forward_to=35'15))", 2024-04-19T02:03:20.116 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.751929+0000", 2024-04-19T02:03:20.116 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.31579944500000001, 2024-04-19T02:03:20.116 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0012116550000000001, 2024-04-19T02:03:20.116 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.116 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.116 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:20.116 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.116 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.116 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.117 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.751929+0000", 2024-04-19T02:03:20.117 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.117 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.117 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.117 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.117 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.751929+0000", 2024-04-19T02:03:20.117 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.117 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.117 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.117 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.117 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.751930+0000", 2024-04-19T02:03:20.117 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.528e-06 2024-04-19T02:03:20.117 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.117 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.117 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.118 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.751946+0000", 2024-04-19T02:03:20.118 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5889999999999999e-05 2024-04-19T02:03:20.118 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.118 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.118 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.118 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.751968+0000", 2024-04-19T02:03:20.118 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1977e-05 2024-04-19T02:03:20.118 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.118 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.118 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.118 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.751982+0000", 2024-04-19T02:03:20.118 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4419e-05 2024-04-19T02:03:20.118 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.118 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.118 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.119 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.752092+0000", 2024-04-19T02:03:20.119 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010935899999999999 2024-04-19T02:03:20.119 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.119 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.119 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:20.119 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.752101+0000", 2024-04-19T02:03:20.119 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.7510000000000003e-06 2024-04-19T02:03:20.119 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.119 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.119 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:20.119 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.753084+0000", 2024-04-19T02:03:20.119 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00098245800000000007 2024-04-19T02:03:20.119 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.119 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.119 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.119 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.753140+0000", 2024-04-19T02:03:20.120 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.6273000000000001e-05 2024-04-19T02:03:20.120 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.120 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.120 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.120 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.120 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.120 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.as7 35/26 ECSubWrite(tid=59, reqid=client.4509.0:285, at_version=35'23, trim_to=0'0, roll_forward_to=34'22))", 2024-04-19T02:03:20.120 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.754512+0000", 2024-04-19T02:03:20.120 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.31321574499999999, 2024-04-19T02:03:20.120 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0031998590000000002, 2024-04-19T02:03:20.120 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.120 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.120 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:20.120 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.120 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.121 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.121 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.754512+0000", 2024-04-19T02:03:20.121 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.121 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.121 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.121 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.121 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.754512+0000", 2024-04-19T02:03:20.121 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.121 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.121 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.121 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.121 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.754514+0000", 2024-04-19T02:03:20.121 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.257e-06 2024-04-19T02:03:20.121 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.121 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.121 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.122 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.754796+0000", 2024-04-19T02:03:20.122 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00028269500000000002 2024-04-19T02:03:20.122 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.122 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.122 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.122 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.754814+0000", 2024-04-19T02:03:20.122 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.774e-05 2024-04-19T02:03:20.122 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.122 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.122 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.122 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.754824+0000", 2024-04-19T02:03:20.122 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.8649999999999996e-06 2024-04-19T02:03:20.122 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.122 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.123 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.123 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.754933+0000", 2024-04-19T02:03:20.123 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108769 2024-04-19T02:03:20.123 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.123 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.123 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:20.123 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.754943+0000", 2024-04-19T02:03:20.123 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0071000000000001e-05 2024-04-19T02:03:20.123 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.123 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.123 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:20.123 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.757669+0000", 2024-04-19T02:03:20.123 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0027265929999999998 2024-04-19T02:03:20.123 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.123 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.123 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.124 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.757712+0000", 2024-04-19T02:03:20.124 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2868999999999999e-05 2024-04-19T02:03:20.124 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.124 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.124 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.124 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.124 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.124 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.as7 35/26 ECSubWrite(tid=60, reqid=client.4509.0:286, at_version=35'24, trim_to=0'0, roll_forward_to=34'22))", 2024-04-19T02:03:20.124 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.758396+0000", 2024-04-19T02:03:20.124 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.30933224799999998, 2024-04-19T02:03:20.124 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0011837519999999999, 2024-04-19T02:03:20.124 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.124 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.124 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:20.124 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.125 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.125 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.125 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.758396+0000", 2024-04-19T02:03:20.125 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.125 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.125 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.125 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.125 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.758396+0000", 2024-04-19T02:03:20.125 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.125 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.125 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.125 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.125 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.758397+0000", 2024-04-19T02:03:20.125 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.85e-07 2024-04-19T02:03:20.125 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.125 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.126 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.126 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.758494+0000", 2024-04-19T02:03:20.126 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.7231999999999993e-05 2024-04-19T02:03:20.126 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.126 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.126 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.126 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.758521+0000", 2024-04-19T02:03:20.126 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7509999999999999e-05 2024-04-19T02:03:20.126 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.126 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.126 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.126 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.758542+0000", 2024-04-19T02:03:20.126 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0531999999999999e-05 2024-04-19T02:03:20.126 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.127 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.127 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.127 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.758683+0000", 2024-04-19T02:03:20.127 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014115500000000001 2024-04-19T02:03:20.127 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.127 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.127 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:20.127 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.758693+0000", 2024-04-19T02:03:20.127 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0414000000000001e-05 2024-04-19T02:03:20.127 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.127 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.127 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:20.127 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.759523+0000", 2024-04-19T02:03:20.127 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00082909599999999998 2024-04-19T02:03:20.127 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.128 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.128 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.128 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.759579+0000", 2024-04-19T02:03:20.128 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.6928000000000001e-05 2024-04-19T02:03:20.128 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.128 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.128 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.128 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.128 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.128 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.as7 35/26 ECSubWrite(tid=61, reqid=client.4509.0:287, at_version=35'25, trim_to=0'0, roll_forward_to=34'22))", 2024-04-19T02:03:20.128 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.760643+0000", 2024-04-19T02:03:20.128 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.307085477, 2024-04-19T02:03:20.128 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00083733799999999997, 2024-04-19T02:03:20.128 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.128 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.128 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:20.129 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.129 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.129 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.129 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.760643+0000", 2024-04-19T02:03:20.129 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.129 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.129 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.129 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.129 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.760643+0000", 2024-04-19T02:03:20.129 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.129 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.129 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.129 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.129 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.760643+0000", 2024-04-19T02:03:20.129 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.6899999999999996e-07 2024-04-19T02:03:20.130 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.130 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.130 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.130 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.760660+0000", 2024-04-19T02:03:20.130 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6656000000000001e-05 2024-04-19T02:03:20.130 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.130 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.130 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.130 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.760675+0000", 2024-04-19T02:03:20.130 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5330000000000001e-05 2024-04-19T02:03:20.130 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.130 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.130 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.130 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.760684+0000", 2024-04-19T02:03:20.130 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.8030000000000003e-06 2024-04-19T02:03:20.131 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.131 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.131 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.131 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.760785+0000", 2024-04-19T02:03:20.131 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000100898 2024-04-19T02:03:20.131 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.131 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.131 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:20.131 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.760794+0000", 2024-04-19T02:03:20.131 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.986e-06 2024-04-19T02:03:20.131 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.131 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.131 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:20.131 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.761442+0000", 2024-04-19T02:03:20.132 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00064790799999999999 2024-04-19T02:03:20.132 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.132 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.132 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.132 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.761480+0000", 2024-04-19T02:03:20.132 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.7888e-05 2024-04-19T02:03:20.132 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.132 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.132 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.132 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.132 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.132 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.as7 35/26 ECSubWrite(tid=62, reqid=client.4509.0:287, at_version=0'0, trim_to=0'0, roll_forward_to=35'25))", 2024-04-19T02:03:20.132 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.764660+0000", 2024-04-19T02:03:20.132 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.303068434, 2024-04-19T02:03:20.132 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00068914400000000004, 2024-04-19T02:03:20.133 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.133 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.133 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:20.133 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.133 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.133 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.133 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.764660+0000", 2024-04-19T02:03:20.133 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.133 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.133 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.133 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.133 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.764660+0000", 2024-04-19T02:03:20.133 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.133 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.133 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.134 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.134 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.764660+0000", 2024-04-19T02:03:20.134 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0299999999999998e-07 2024-04-19T02:03:20.134 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.134 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.134 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.134 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.764673+0000", 2024-04-19T02:03:20.134 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2921e-05 2024-04-19T02:03:20.134 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.134 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.134 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.134 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.764705+0000", 2024-04-19T02:03:20.134 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1628999999999998e-05 2024-04-19T02:03:20.134 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.135 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.135 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.135 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.764715+0000", 2024-04-19T02:03:20.135 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.7489999999999993e-06 2024-04-19T02:03:20.135 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.135 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.135 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.135 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.764779+0000", 2024-04-19T02:03:20.135 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.4617999999999997e-05 2024-04-19T02:03:20.135 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.135 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.135 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:20.135 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.764790+0000", 2024-04-19T02:03:20.135 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.116e-05 2024-04-19T02:03:20.135 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.136 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.136 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:20.136 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.765313+0000", 2024-04-19T02:03:20.136 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00052242200000000003 2024-04-19T02:03:20.136 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.136 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.136 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.136 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.765349+0000", 2024-04-19T02:03:20.136 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5842e-05 2024-04-19T02:03:20.136 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.136 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.136 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.136 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.136 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.136 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs2 35/26 ECSubWrite(tid=64, reqid=client.4509.0:290, at_version=35'17, trim_to=0'0, roll_forward_to=35'16))", 2024-04-19T02:03:20.137 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.785126+0000", 2024-04-19T02:03:20.137 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.28260160400000001, 2024-04-19T02:03:20.137 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00094724900000000005, 2024-04-19T02:03:20.137 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.137 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.137 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:20.137 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.137 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.137 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.137 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.785126+0000", 2024-04-19T02:03:20.137 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.137 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.137 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.137 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.138 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.785126+0000", 2024-04-19T02:03:20.138 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.138 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.138 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.138 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.138 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.785127+0000", 2024-04-19T02:03:20.138 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5300000000000003e-07 2024-04-19T02:03:20.138 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.138 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.138 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.138 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.785145+0000", 2024-04-19T02:03:20.138 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7906000000000001e-05 2024-04-19T02:03:20.138 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.138 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.138 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.139 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.785157+0000", 2024-04-19T02:03:20.139 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1634000000000001e-05 2024-04-19T02:03:20.139 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.139 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.139 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.139 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.785166+0000", 2024-04-19T02:03:20.139 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.8510000000000005e-06 2024-04-19T02:03:20.139 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.139 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.139 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.139 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.785249+0000", 2024-04-19T02:03:20.139 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3712000000000001e-05 2024-04-19T02:03:20.139 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.139 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.140 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:20.140 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.785257+0000", 2024-04-19T02:03:20.140 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.8650000000000001e-06 2024-04-19T02:03:20.140 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.140 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.140 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:20.140 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.786028+0000", 2024-04-19T02:03:20.140 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000771001 2024-04-19T02:03:20.140 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.140 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.140 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.140 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.786074+0000", 2024-04-19T02:03:20.140 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5426999999999997e-05 2024-04-19T02:03:20.140 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.140 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.140 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.141 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.141 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.141 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs2 35/26 ECSubWrite(tid=65, reqid=client.4509.0:290, at_version=0'0, trim_to=0'0, roll_forward_to=35'17))", 2024-04-19T02:03:20.141 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.788530+0000", 2024-04-19T02:03:20.141 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.27919818299999999, 2024-04-19T02:03:20.141 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00079237700000000003, 2024-04-19T02:03:20.141 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.141 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.141 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:20.141 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.141 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.141 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.141 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.788530+0000", 2024-04-19T02:03:20.141 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.141 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.142 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.142 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.142 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.788530+0000", 2024-04-19T02:03:20.142 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.142 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.142 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.142 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.142 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.788531+0000", 2024-04-19T02:03:20.142 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.7800000000000001e-07 2024-04-19T02:03:20.142 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.142 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.142 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.142 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.788541+0000", 2024-04-19T02:03:20.142 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0533000000000001e-05 2024-04-19T02:03:20.142 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.143 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.143 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.143 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.788554+0000", 2024-04-19T02:03:20.143 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3088000000000001e-05 2024-04-19T02:03:20.143 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.143 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.143 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.143 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.788565+0000", 2024-04-19T02:03:20.143 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0869e-05 2024-04-19T02:03:20.143 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.143 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.143 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.143 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.788657+0000", 2024-04-19T02:03:20.143 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.1503999999999997e-05 2024-04-19T02:03:20.143 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.144 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.144 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:20.144 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.788665+0000", 2024-04-19T02:03:20.144 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7299999999999994e-06 2024-04-19T02:03:20.144 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.144 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.144 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:20.144 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.789293+0000", 2024-04-19T02:03:20.144 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00062724000000000005 2024-04-19T02:03:20.144 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.144 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.144 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.144 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.789322+0000", 2024-04-19T02:03:20.144 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9635e-05 2024-04-19T02:03:20.144 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.145 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.145 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.145 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.145 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.145 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:20.147 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-19T02:03:20.147 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:20.147+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:20.247 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:20.247+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:20.313 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:20.313 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:03:20.313 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-04-19T02:03:20.313 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:20.327 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-19T02:03:20.350 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:20.351+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:20.436 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:20.436 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:03:20.436 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-04-19T02:03:20.436 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-04-19T02:03:20.437 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:20.449 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-19T02:03:20.451 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:20.451+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:20.554 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:20.555+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:20.566 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:20.566 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-04-19T02:03:20.566 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-04-19T02:03:20.566 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-04-19T02:03:20.566 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.566 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:230 3.2s0 3:43ce42f3:::smithi05930959-27:head [stat,delete] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-19T02:03:20.566 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:15.635171+0000", 2024-04-19T02:03:20.566 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 4.9319433699999999, 2024-04-19T02:03:20.566 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.013910249, 2024-04-19T02:03:20.566 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.566 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.566 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:20.567 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:20.567 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:20.567 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:20.567 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 230 2024-04-19T02:03:20.567 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.567 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.567 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.567 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.567 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.635171+0000", 2024-04-19T02:03:20.567 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.567 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.567 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.567 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.567 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.635171+0000", 2024-04-19T02:03:20.568 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.568 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.568 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.568 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.568 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.635175+0000", 2024-04-19T02:03:20.568 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2399999999999999e-06 2024-04-19T02:03:20.568 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.568 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.568 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.568 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.635190+0000", 2024-04-19T02:03:20.568 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.517e-05 2024-04-19T02:03:20.568 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.568 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.568 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.569 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.635226+0000", 2024-04-19T02:03:20.569 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6495999999999998e-05 2024-04-19T02:03:20.569 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.569 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.569 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.569 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.635250+0000", 2024-04-19T02:03:20.569 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3713000000000001e-05 2024-04-19T02:03:20.569 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.569 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.569 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.569 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.635429+0000", 2024-04-19T02:03:20.569 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00017871599999999999 2024-04-19T02:03:20.569 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.569 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.570 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "op must wait for map", 2024-04-19T02:03:20.570 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.635456+0000", 2024-04-19T02:03:20.570 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7373e-05 2024-04-19T02:03:20.570 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.570 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.570 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.570 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.639506+0000", 2024-04-19T02:03:20.570 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0040497620000000002 2024-04-19T02:03:20.570 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.570 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.570 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:20.570 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.639758+0000", 2024-04-19T02:03:20.570 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00025186499999999999 2024-04-19T02:03:20.570 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.571 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.571 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:20.571 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.641987+0000", 2024-04-19T02:03:20.571 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0022297290000000002 2024-04-19T02:03:20.571 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.571 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.571 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:20.571 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.644866+0000", 2024-04-19T02:03:20.571 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0028787550000000002 2024-04-19T02:03:20.571 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.571 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.571 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:20.571 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.648961+0000", 2024-04-19T02:03:20.571 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0040949979999999999 2024-04-19T02:03:20.571 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.572 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.572 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.572 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.649082+0000", 2024-04-19T02:03:20.572 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000120432 2024-04-19T02:03:20.572 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.572 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.572 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.572 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.572 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.572 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:231 3.2s0 3:439cb4d1:::smithi05930959-23:head [append 0~671744] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-19T02:03:20.572 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:15.657794+0000", 2024-04-19T02:03:20.572 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 4.9093210709999999, 2024-04-19T02:03:20.572 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012024274, 2024-04-19T02:03:20.572 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.572 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.573 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:20.573 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:20.573 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:20.573 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:20.573 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 231 2024-04-19T02:03:20.573 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.573 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.573 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.573 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.573 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.657794+0000", 2024-04-19T02:03:20.573 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.573 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.573 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.573 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.573 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.657794+0000", 2024-04-19T02:03:20.574 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.574 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.574 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.574 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.574 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.657797+0000", 2024-04-19T02:03:20.574 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.23e-06 2024-04-19T02:03:20.574 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.574 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.574 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.574 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.658419+0000", 2024-04-19T02:03:20.574 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00062175499999999999 2024-04-19T02:03:20.574 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.574 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.574 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.575 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.658453+0000", 2024-04-19T02:03:20.575 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4116999999999999e-05 2024-04-19T02:03:20.575 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.575 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.575 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.575 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.658475+0000", 2024-04-19T02:03:20.575 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2203000000000001e-05 2024-04-19T02:03:20.575 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.575 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.575 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.575 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.658611+0000", 2024-04-19T02:03:20.575 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000136425 2024-04-19T02:03:20.575 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.575 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.575 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:20.576 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.658797+0000", 2024-04-19T02:03:20.576 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00018583599999999999 2024-04-19T02:03:20.576 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.576 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.576 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:20.576 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661233+0000", 2024-04-19T02:03:20.576 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0024353869999999998 2024-04-19T02:03:20.576 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.576 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.576 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:20.576 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.668947+0000", 2024-04-19T02:03:20.576 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0077140139999999999 2024-04-19T02:03:20.576 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.576 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.576 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:20.577 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.669636+0000", 2024-04-19T02:03:20.577 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00068981399999999997 2024-04-19T02:03:20.577 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.577 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.577 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.577 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.669818+0000", 2024-04-19T02:03:20.577 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00018149299999999999 2024-04-19T02:03:20.577 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.577 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.577 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.577 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.577 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.577 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:232 3.2s0 3:439cb4d1:::smithi05930959-23:head [append 0~376832] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-19T02:03:20.577 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:15.660690+0000", 2024-04-19T02:03:20.577 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 4.9064242169999996, 2024-04-19T02:03:20.578 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010153852, 2024-04-19T02:03:20.578 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.578 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.578 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:20.578 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:20.578 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:20.578 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:20.578 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 232 2024-04-19T02:03:20.578 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.578 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.578 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.578 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.578 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.660690+0000", 2024-04-19T02:03:20.578 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.579 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.579 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.579 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.579 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.660690+0000", 2024-04-19T02:03:20.579 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.579 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.579 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.579 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.579 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.660693+0000", 2024-04-19T02:03:20.579 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.903e-06 2024-04-19T02:03:20.579 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.579 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.579 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.579 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661327+0000", 2024-04-19T02:03:20.580 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00063362200000000003 2024-04-19T02:03:20.580 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.580 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.580 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.580 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661402+0000", 2024-04-19T02:03:20.580 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.5190000000000003e-05 2024-04-19T02:03:20.580 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.580 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.580 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.580 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661426+0000", 2024-04-19T02:03:20.580 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3519e-05 2024-04-19T02:03:20.580 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.580 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.580 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.580 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.663292+0000", 2024-04-19T02:03:20.581 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0018659989999999999 2024-04-19T02:03:20.581 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.581 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.581 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:20.581 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.663512+0000", 2024-04-19T02:03:20.581 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000220118 2024-04-19T02:03:20.581 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.581 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.581 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:20.581 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.665317+0000", 2024-04-19T02:03:20.581 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0018056260000000001 2024-04-19T02:03:20.581 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.581 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.581 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:20.582 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.668949+0000", 2024-04-19T02:03:20.582 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.003631563 2024-04-19T02:03:20.582 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.582 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.582 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:20.582 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.670735+0000", 2024-04-19T02:03:20.582 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001786096 2024-04-19T02:03:20.582 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.582 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.582 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.582 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.670844+0000", 2024-04-19T02:03:20.582 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010921600000000001 2024-04-19T02:03:20.583 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.583 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.583 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.583 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.583 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.583 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:233 3.2s0 3:439cb4d1:::smithi05930959-23:head [setxattr _header (56) in=7b] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-19T02:03:20.583 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:15.661479+0000", 2024-04-19T02:03:20.583 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 4.9056355429999998, 2024-04-19T02:03:20.583 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010065869999999999, 2024-04-19T02:03:20.583 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.583 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.583 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:20.583 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:20.583 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:20.584 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:20.584 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 233 2024-04-19T02:03:20.584 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.584 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.584 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.584 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.584 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661479+0000", 2024-04-19T02:03:20.584 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.584 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.584 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.584 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.584 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661479+0000", 2024-04-19T02:03:20.584 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.584 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.584 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.585 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.585 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661481+0000", 2024-04-19T02:03:20.585 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.72e-06 2024-04-19T02:03:20.585 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.585 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.585 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.585 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661493+0000", 2024-04-19T02:03:20.585 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1653e-05 2024-04-19T02:03:20.585 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.585 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.585 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.585 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661517+0000", 2024-04-19T02:03:20.585 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4749999999999999e-05 2024-04-19T02:03:20.585 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.585 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.586 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.586 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661535+0000", 2024-04-19T02:03:20.586 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8003e-05 2024-04-19T02:03:20.586 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.586 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.586 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.586 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.666609+0000", 2024-04-19T02:03:20.586 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.005073691 2024-04-19T02:03:20.586 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.586 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.586 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:20.586 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.666880+0000", 2024-04-19T02:03:20.586 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00027123099999999998 2024-04-19T02:03:20.587 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.587 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.587 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:20.587 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.668034+0000", 2024-04-19T02:03:20.587 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001153421 2024-04-19T02:03:20.587 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.587 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.587 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:20.587 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.670252+0000", 2024-04-19T02:03:20.587 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002218397 2024-04-19T02:03:20.587 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.587 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.587 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:20.587 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.671448+0000", 2024-04-19T02:03:20.588 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0011963189999999999 2024-04-19T02:03:20.588 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.588 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.588 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.588 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.671545+0000", 2024-04-19T02:03:20.588 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.6685e-05 2024-04-19T02:03:20.588 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.588 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.588 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.588 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.588 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.588 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:234 3.2s0 3:439cb4d1:::smithi05930959-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-19T02:03:20.588 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:15.661605+0000", 2024-04-19T02:03:20.588 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 4.9055092570000003, 2024-04-19T02:03:20.588 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.013322123, 2024-04-19T02:03:20.589 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.589 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.589 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:20.589 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:20.589 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:20.589 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:20.589 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 234 2024-04-19T02:03:20.589 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.589 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.589 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.589 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.589 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661605+0000", 2024-04-19T02:03:20.589 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.589 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.590 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.590 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.590 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661605+0000", 2024-04-19T02:03:20.590 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.590 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.590 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.590 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.590 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661607+0000", 2024-04-19T02:03:20.590 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5600000000000001e-06 2024-04-19T02:03:20.590 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.590 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.590 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.590 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661617+0000", 2024-04-19T02:03:20.590 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0436e-05 2024-04-19T02:03:20.590 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.591 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.591 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.591 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661644+0000", 2024-04-19T02:03:20.591 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6376999999999998e-05 2024-04-19T02:03:20.591 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.591 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.591 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.591 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661662+0000", 2024-04-19T02:03:20.591 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7710000000000002e-05 2024-04-19T02:03:20.591 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.591 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.591 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.591 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.672340+0000", 2024-04-19T02:03:20.591 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010678974000000001 2024-04-19T02:03:20.591 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.592 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.592 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:20.592 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.672598+0000", 2024-04-19T02:03:20.592 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00025784799999999997 2024-04-19T02:03:20.592 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.592 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.592 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.592 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.674928+0000", 2024-04-19T02:03:20.592 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0023292180000000001 2024-04-19T02:03:20.592 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.593 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.593 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.593 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.593 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.593 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 31/26 ECSubWriteReply(tid=67, last_complete=31'26, committed=1, applied=1))", 2024-04-19T02:03:20.593 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:15.663432+0000", 2024-04-19T02:03:20.593 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 4.9036827110000001, 2024-04-19T02:03:20.593 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0053768610000000001, 2024-04-19T02:03:20.593 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.593 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.593 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:20.593 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.594 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.594 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.594 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.663432+0000", 2024-04-19T02:03:20.594 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.594 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.594 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.594 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.594 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.663432+0000", 2024-04-19T02:03:20.594 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.594 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.594 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.594 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.594 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.663434+0000", 2024-04-19T02:03:20.595 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0099999999999998e-06 2024-04-19T02:03:20.595 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.595 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.595 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.595 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.663447+0000", 2024-04-19T02:03:20.595 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.34e-05 2024-04-19T02:03:20.595 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.595 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.595 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.595 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.663476+0000", 2024-04-19T02:03:20.595 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8651999999999999e-05 2024-04-19T02:03:20.595 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.595 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.595 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.596 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.663494+0000", 2024-04-19T02:03:20.596 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8216999999999999e-05 2024-04-19T02:03:20.596 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.596 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.596 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.596 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.668760+0000", 2024-04-19T02:03:20.596 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0052659330000000004 2024-04-19T02:03:20.596 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.596 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.596 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.596 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.668809+0000", 2024-04-19T02:03:20.596 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.8649000000000001e-05 2024-04-19T02:03:20.596 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.596 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.596 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.597 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.597 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.597 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:236 3.2s0 3:4aa53487:::smithi05930959-17:head [create,setxattr pUpejN\\XeR[ScpqbhnaXajbqbVcs]\\YUaoVqmdSQ (16807) in=40b,setxattr ^TXcXY\\i\\UaegqjXsVZRnYhXQcQtNan^ZkmbhgfY (349) in=40b,setxattr ekSnl]Q^sZQXVqUlaf`k_aifQmosa`q\\TjqRt_Qf (1227) in=40b,setxattr \\iWlX_lYknVg`YeraZ]fTRtp[SmcmU\\fi`W]WrbQ (6244) in=40b,setxattr jXafahNhrkaS]`RTUfZsViWZQb_fe[hNa`eaSXaP (104) in=40b,setxattr SQfdcs\\p[W^UXTPlNX^ZpfRc`f[oTemUqrW]noj\\ (1227) in=40b,setxattr \\^nWTlXbgpUbdWkgbnaQn`^hmQ[]WmV]lU[SaXU] (1542) in=40b,setxattr dVRspsbTY\\[_ajteqtXtW[`oNOrUtVnqhPan_ePQ (314) in=40b,setxattr tQcXlPoWcYqTci^srZesYV[VgVaRshSrZtc\\mZa^ (67) in=40b,setxattr s]QeT_cN\\hZPf_Whl\\VZhhldY`gN^hNQ[rbdZ\\aZ (244) in=40b,setxattr Sga\\QYQOYspmkSbrWjWZXUeqopRefVUPXn]SjYcY (1719) in=40b,setxattr [NiqOQcWdfT`oVaqXYf_dm`h_g]OscjStsShYaUN (876) in=40b,setxattr Ok_\\_emNX_rOSq`pl[ijceSO_WNtXVhjhtTPolkN (1647) in=40b,setxattr saRsYUrZhdhmY^k^i[Rph][cqVtPsi`WarjOnYO^ (1155) in=40b,setxattr cY\\acssfT[QaU\\pigijbXs\\ROcY__s[R_geknqaa (1472) in=40b,setxattr WfSmRUdrW[ojWRr_kTre\\ns`t[cOrTVTNankd[OP (350) in=40b,setxattr T`ce]ZdmspasZU]X[c[socq[elaihoTSQi_WieWt (1226) in=40b,setxattr piRXhWpPba^f_P_Z[\\`\\Ugj[kRno\\VgbWQlndVa] (1718) in=40b,setxattr lV]d\\p[Pbb_ShS]kXZmPXqpq`QhRa`cXeNpTNciq (875) in=40b,setxattr cdtSkTmQRaR]lfSnj]qi[pkO`[NRbaqUWsnj]XiO (314) in=40b] snapc 3=[2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-19T02:03:20.597 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.646243+0000", 2024-04-19T02:03:20.597 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 3.9208718820000001, 2024-04-19T02:03:20.597 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.022457007000000001, 2024-04-19T02:03:20.597 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.597 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.597 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:20.597 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:20.597 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:20.597 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:20.597 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 236 2024-04-19T02:03:20.597 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.598 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.598 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.598 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.598 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646243+0000", 2024-04-19T02:03:20.598 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.598 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.598 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.598 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.598 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646243+0000", 2024-04-19T02:03:20.598 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.598 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.598 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.598 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.598 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646246+0000", 2024-04-19T02:03:20.599 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.92e-06 2024-04-19T02:03:20.599 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.599 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.599 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.599 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646351+0000", 2024-04-19T02:03:20.599 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000105393 2024-04-19T02:03:20.599 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.599 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.599 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.599 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646385+0000", 2024-04-19T02:03:20.599 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4326000000000003e-05 2024-04-19T02:03:20.599 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.599 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.599 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.599 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646403+0000", 2024-04-19T02:03:20.600 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7543000000000001e-05 2024-04-19T02:03:20.600 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.600 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.600 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.600 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646534+0000", 2024-04-19T02:03:20.600 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013077600000000001 2024-04-19T02:03:20.600 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.600 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.600 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "op must wait for map", 2024-04-19T02:03:20.600 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646600+0000", 2024-04-19T02:03:20.600 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.5977999999999997e-05 2024-04-19T02:03:20.600 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.600 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.600 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.600 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.650956+0000", 2024-04-19T02:03:20.601 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0043565890000000001 2024-04-19T02:03:20.601 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.601 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.601 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:20.601 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.651213+0000", 2024-04-19T02:03:20.601 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00025659899999999999 2024-04-19T02:03:20.601 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.601 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.601 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:20.601 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.655246+0000", 2024-04-19T02:03:20.601 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.004033549 2024-04-19T02:03:20.601 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.601 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.601 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:20.601 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.668007+0000", 2024-04-19T02:03:20.602 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012760535 2024-04-19T02:03:20.602 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.602 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.602 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:20.602 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.668552+0000", 2024-04-19T02:03:20.602 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00054500000000000002 2024-04-19T02:03:20.602 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.602 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.602 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.602 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.668700+0000", 2024-04-19T02:03:20.602 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000147799 2024-04-19T02:03:20.602 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.602 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.602 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.603 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.603 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.603 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:237 3.2s0 3:407ecba7:::smithi05930959-22:2 [read 0~360448,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-19T02:03:20.603 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.646438+0000", 2024-04-19T02:03:20.603 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 3.9206762250000002, 2024-04-19T02:03:20.603 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.024327603, 2024-04-19T02:03:20.603 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.603 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.603 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:20.603 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:20.603 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:20.603 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:20.603 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 237 2024-04-19T02:03:20.603 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.604 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.604 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.604 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.604 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646438+0000", 2024-04-19T02:03:20.604 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.604 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.604 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.604 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.604 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646438+0000", 2024-04-19T02:03:20.604 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.604 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.604 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.604 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.604 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646440+0000", 2024-04-19T02:03:20.604 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.254e-06 2024-04-19T02:03:20.605 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.605 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.605 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.605 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646447+0000", 2024-04-19T02:03:20.605 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.3289999999999996e-06 2024-04-19T02:03:20.605 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.605 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.605 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.605 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646463+0000", 2024-04-19T02:03:20.605 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5846e-05 2024-04-19T02:03:20.605 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.605 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.605 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.605 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646472+0000", 2024-04-19T02:03:20.606 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.2939999999999994e-06 2024-04-19T02:03:20.606 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.606 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.606 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.606 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646953+0000", 2024-04-19T02:03:20.606 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000480901 2024-04-19T02:03:20.606 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.606 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.606 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:03:20.606 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646971+0000", 2024-04-19T02:03:20.606 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8167000000000001e-05 2024-04-19T02:03:20.606 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.606 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.607 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.607 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.656667+0000", 2024-04-19T02:03:20.607 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0096952449999999999 2024-04-19T02:03:20.607 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.607 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.607 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:20.607 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.656833+0000", 2024-04-19T02:03:20.607 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00016697500000000001 2024-04-19T02:03:20.607 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.607 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.607 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.607 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.670766+0000", 2024-04-19T02:03:20.607 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.013932592000000001 2024-04-19T02:03:20.607 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.608 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.608 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.608 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.608 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.608 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:238 3.2s0 3:407ecba7:::smithi05930959-22:2 [read 0~360448,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-19T02:03:20.608 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.646510+0000", 2024-04-19T02:03:20.608 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 3.9206049429999998, 2024-04-19T02:03:20.608 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.028744658999999999, 2024-04-19T02:03:20.608 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.608 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.608 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:20.608 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:20.608 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:20.608 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:20.608 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 238 2024-04-19T02:03:20.609 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.609 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.609 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.609 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.609 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646510+0000", 2024-04-19T02:03:20.609 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.609 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.609 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.609 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.609 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646510+0000", 2024-04-19T02:03:20.609 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.609 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.609 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.609 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.610 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646511+0000", 2024-04-19T02:03:20.610 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.8599999999999997e-07 2024-04-19T02:03:20.610 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.610 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.610 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.610 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646517+0000", 2024-04-19T02:03:20.610 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.1700000000000002e-06 2024-04-19T02:03:20.610 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.610 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.610 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.610 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646528+0000", 2024-04-19T02:03:20.610 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1202999999999999e-05 2024-04-19T02:03:20.610 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.610 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.611 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.611 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646537+0000", 2024-04-19T02:03:20.611 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7460000000000006e-06 2024-04-19T02:03:20.611 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.611 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.611 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.611 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.647105+0000", 2024-04-19T02:03:20.611 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00056801599999999996 2024-04-19T02:03:20.611 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.611 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.611 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:03:20.611 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.647127+0000", 2024-04-19T02:03:20.611 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.175e-05 2024-04-19T02:03:20.611 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.612 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.612 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.612 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.657504+0000", 2024-04-19T02:03:20.612 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010377961 2024-04-19T02:03:20.612 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.612 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.612 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:20.612 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.657826+0000", 2024-04-19T02:03:20.612 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00032128300000000001 2024-04-19T02:03:20.612 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.612 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.612 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.612 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.675254+0000", 2024-04-19T02:03:20.612 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.017428644 2024-04-19T02:03:20.612 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.613 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.613 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.613 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.613 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.613 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:239 3.2s0 3:407ecba7:::smithi05930959-22:2 [read 0~360448,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-19T02:03:20.613 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.646549+0000", 2024-04-19T02:03:20.613 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 3.9205654719999998, 2024-04-19T02:03:20.613 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.029381570999999999, 2024-04-19T02:03:20.613 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.613 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.613 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:20.613 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:20.613 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:20.613 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:20.614 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 239 2024-04-19T02:03:20.614 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.614 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.614 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.614 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.614 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646549+0000", 2024-04-19T02:03:20.614 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.614 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.614 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.614 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.614 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646549+0000", 2024-04-19T02:03:20.614 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.614 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.614 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.615 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.615 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646550+0000", 2024-04-19T02:03:20.615 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8400000000000004e-07 2024-04-19T02:03:20.615 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.615 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.615 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.615 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646555+0000", 2024-04-19T02:03:20.615 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9989999999999999e-06 2024-04-19T02:03:20.615 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.615 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.615 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.615 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646567+0000", 2024-04-19T02:03:20.615 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.202e-05 2024-04-19T02:03:20.615 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.615 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.616 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.616 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646575+0000", 2024-04-19T02:03:20.616 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.7640000000000005e-06 2024-04-19T02:03:20.616 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.616 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.616 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.616 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.647229+0000", 2024-04-19T02:03:20.616 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00065415299999999998 2024-04-19T02:03:20.616 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.616 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.616 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:03:20.616 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.647255+0000", 2024-04-19T02:03:20.616 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5795999999999998e-05 2024-04-19T02:03:20.616 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.616 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.617 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.617 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.659835+0000", 2024-04-19T02:03:20.617 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012580713 2024-04-19T02:03:20.617 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.617 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.617 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:20.617 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.660090+0000", 2024-04-19T02:03:20.617 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00025430899999999998 2024-04-19T02:03:20.617 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.617 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.617 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.617 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.675931+0000", 2024-04-19T02:03:20.617 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.015841133 2024-04-19T02:03:20.617 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.617 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.618 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.618 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.618 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.618 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpRead(3.2s0 32/26 ECSubRead(tid=73, to_read={3:407ecba7:::smithi05930959-22:head=0,90112,0}, subchunks={3:407ecba7:::smithi05930959-22:head=[0,1]}, attrs_to_read=))", 2024-04-19T02:03:20.618 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.657193+0000", 2024-04-19T02:03:20.618 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 3.9099212990000001, 2024-04-19T02:03:20.618 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0058328010000000003, 2024-04-19T02:03:20.618 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.618 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.618 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:20.618 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.618 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.618 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.619 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.657193+0000", 2024-04-19T02:03:20.619 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.619 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.619 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.619 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.619 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.657193+0000", 2024-04-19T02:03:20.619 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.619 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.619 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.619 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.619 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.657193+0000", 2024-04-19T02:03:20.619 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.619 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.619 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.620 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.620 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.657193+0000", 2024-04-19T02:03:20.620 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.620 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.620 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.620 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.620 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.657246+0000", 2024-04-19T02:03:20.620 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.2191999999999998e-05 2024-04-19T02:03:20.620 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.620 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.620 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.620 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.657275+0000", 2024-04-19T02:03:20.620 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9776000000000001e-05 2024-04-19T02:03:20.620 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.621 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.621 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.621 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.662061+0000", 2024-04-19T02:03:20.621 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.004785816 2024-04-19T02:03:20.621 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.621 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.621 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.621 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.663026+0000", 2024-04-19T02:03:20.621 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00096501699999999998 2024-04-19T02:03:20.621 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.621 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.621 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.621 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.621 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.622 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpRead(3.2s0 32/26 ECSubRead(tid=74, to_read={3:407ecba7:::smithi05930959-22:head=0,90112,0}, subchunks={3:407ecba7:::smithi05930959-22:head=[0,1]}, attrs_to_read=))", 2024-04-19T02:03:20.622 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.658518+0000", 2024-04-19T02:03:20.622 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 3.9085967859999999, 2024-04-19T02:03:20.622 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0054588589999999999, 2024-04-19T02:03:20.622 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.622 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.622 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:20.622 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.622 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.622 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.622 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.658518+0000", 2024-04-19T02:03:20.622 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.622 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.622 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.623 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.623 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.658518+0000", 2024-04-19T02:03:20.623 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.623 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.623 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.623 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.623 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.658518+0000", 2024-04-19T02:03:20.623 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.623 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.623 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.623 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.623 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.658518+0000", 2024-04-19T02:03:20.623 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.623 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.623 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.624 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.624 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.658569+0000", 2024-04-19T02:03:20.624 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.1232e-05 2024-04-19T02:03:20.624 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.624 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.624 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.624 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.658596+0000", 2024-04-19T02:03:20.624 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6908999999999999e-05 2024-04-19T02:03:20.624 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.624 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.624 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.624 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.663765+0000", 2024-04-19T02:03:20.624 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0051687479999999999 2024-04-19T02:03:20.624 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.625 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.625 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.625 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.663977+0000", 2024-04-19T02:03:20.625 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00021196999999999999 2024-04-19T02:03:20.625 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.625 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.625 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.625 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.625 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.625 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpRead(3.2s0 32/26 ECSubRead(tid=75, to_read={3:407ecba7:::smithi05930959-22:head=0,90112,0}, subchunks={3:407ecba7:::smithi05930959-22:head=[0,1]}, attrs_to_read=))", 2024-04-19T02:03:20.625 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.660621+0000", 2024-04-19T02:03:20.625 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 3.906493411, 2024-04-19T02:03:20.625 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0067211509999999999, 2024-04-19T02:03:20.625 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.625 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.626 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:20.626 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.626 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.626 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.626 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.660621+0000", 2024-04-19T02:03:20.626 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.626 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.626 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.626 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.626 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.660621+0000", 2024-04-19T02:03:20.626 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.626 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.626 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.626 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.626 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.660621+0000", 2024-04-19T02:03:20.627 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.627 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.627 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.627 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.627 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.660621+0000", 2024-04-19T02:03:20.627 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.627 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.627 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.627 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.627 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.660667+0000", 2024-04-19T02:03:20.627 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.6139e-05 2024-04-19T02:03:20.627 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.627 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.627 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.627 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.660695+0000", 2024-04-19T02:03:20.628 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7093e-05 2024-04-19T02:03:20.628 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.628 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.628 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.628 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.664582+0000", 2024-04-19T02:03:20.628 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0038879299999999999 2024-04-19T02:03:20.628 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.628 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.628 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.628 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.667342+0000", 2024-04-19T02:03:20.628 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002759989 2024-04-19T02:03:20.628 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.628 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.628 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.628 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.629 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.629 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.2s0 32/26 ECSubReadReply(tid=75, attrs_read=0))", 2024-04-19T02:03:20.629 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.661256+0000", 2024-04-19T02:03:20.629 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 3.9058587830000002, 2024-04-19T02:03:20.629 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0067562519999999999, 2024-04-19T02:03:20.629 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.629 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.629 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:20.629 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.629 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.629 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.629 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.661256+0000", 2024-04-19T02:03:20.629 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.629 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.629 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.630 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.630 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.661256+0000", 2024-04-19T02:03:20.630 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.630 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.630 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.630 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.630 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.661286+0000", 2024-04-19T02:03:20.630 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9921999999999999e-05 2024-04-19T02:03:20.630 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.630 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.630 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.630 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.661468+0000", 2024-04-19T02:03:20.630 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000181884 2024-04-19T02:03:20.630 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.631 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.631 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.631 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.661503+0000", 2024-04-19T02:03:20.631 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5685999999999997e-05 2024-04-19T02:03:20.631 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.631 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.631 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.631 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.661521+0000", 2024-04-19T02:03:20.631 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7728999999999999e-05 2024-04-19T02:03:20.631 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.631 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.631 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.631 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.667883+0000", 2024-04-19T02:03:20.631 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.006362075 2024-04-19T02:03:20.631 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.632 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.632 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.632 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.668012+0000", 2024-04-19T02:03:20.632 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000128956 2024-04-19T02:03:20.632 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.632 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.632 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.632 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.632 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.632 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.2s0 32/26 ECSubReadReply(tid=74, attrs_read=0))", 2024-04-19T02:03:20.632 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.662270+0000", 2024-04-19T02:03:20.632 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 3.9048447880000001, 2024-04-19T02:03:20.632 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0060188150000000003, 2024-04-19T02:03:20.632 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.633 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.633 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:20.633 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.633 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.633 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.633 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.662270+0000", 2024-04-19T02:03:20.633 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.633 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.633 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.633 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.633 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.662270+0000", 2024-04-19T02:03:20.633 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.633 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.633 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.634 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.634 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.662271+0000", 2024-04-19T02:03:20.634 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.283e-06 2024-04-19T02:03:20.634 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.634 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.634 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.634 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.662462+0000", 2024-04-19T02:03:20.634 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00019127799999999999 2024-04-19T02:03:20.634 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.634 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.634 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.634 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.662486+0000", 2024-04-19T02:03:20.634 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3756000000000001e-05 2024-04-19T02:03:20.634 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.635 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.635 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.635 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.662501+0000", 2024-04-19T02:03:20.635 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5092000000000001e-05 2024-04-19T02:03:20.635 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.635 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.635 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.635 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.668160+0000", 2024-04-19T02:03:20.635 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0056584799999999996 2024-04-19T02:03:20.635 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.635 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.635 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.635 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.668289+0000", 2024-04-19T02:03:20.635 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00012892599999999999 2024-04-19T02:03:20.635 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.636 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.636 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.636 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.636 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.636 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.2s0 32/26 ECSubReadReply(tid=73, attrs_read=0))", 2024-04-19T02:03:20.636 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.662994+0000", 2024-04-19T02:03:20.636 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 3.9041211799999997, 2024-04-19T02:03:20.636 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0078037799999999997, 2024-04-19T02:03:20.636 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.636 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.636 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:20.636 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.637 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.637 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.637 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.662994+0000", 2024-04-19T02:03:20.637 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.637 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.637 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.637 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.637 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.662994+0000", 2024-04-19T02:03:20.637 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.637 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.637 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.637 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.637 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.662994+0000", 2024-04-19T02:03:20.637 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.638 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.638 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.638 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.638 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.662994+0000", 2024-04-19T02:03:20.638 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.638 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.638 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.638 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.638 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.663041+0000", 2024-04-19T02:03:20.638 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.7849000000000002e-05 2024-04-19T02:03:20.638 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.638 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.638 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.638 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.663079+0000", 2024-04-19T02:03:20.638 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.7832e-05 2024-04-19T02:03:20.639 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.639 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.639 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.639 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.669809+0000", 2024-04-19T02:03:20.639 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0067298530000000001 2024-04-19T02:03:20.639 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.639 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.639 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.639 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.670797+0000", 2024-04-19T02:03:20.639 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00098824599999999992 2024-04-19T02:03:20.639 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.639 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.639 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.639 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.639 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.640 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.2s0 32/26 ECSubReadReply(tid=75, attrs_read=0))", 2024-04-19T02:03:20.640 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.663343+0000", 2024-04-19T02:03:20.640 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 3.9037719109999998, 2024-04-19T02:03:20.640 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.011502955, 2024-04-19T02:03:20.640 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.640 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.640 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:20.640 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.640 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.640 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.640 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.663343+0000", 2024-04-19T02:03:20.640 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.640 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.640 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.640 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.641 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.663343+0000", 2024-04-19T02:03:20.641 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.641 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.641 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.641 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.641 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.663345+0000", 2024-04-19T02:03:20.641 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.153e-06 2024-04-19T02:03:20.641 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.641 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.641 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.641 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.663576+0000", 2024-04-19T02:03:20.641 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000231103 2024-04-19T02:03:20.641 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.641 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.641 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.642 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.663601+0000", 2024-04-19T02:03:20.642 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5113000000000002e-05 2024-04-19T02:03:20.642 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.642 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.642 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.642 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.663618+0000", 2024-04-19T02:03:20.642 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6362999999999999e-05 2024-04-19T02:03:20.642 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.642 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.642 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.642 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.674769+0000", 2024-04-19T02:03:20.642 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.011151978999999999 2024-04-19T02:03:20.642 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.642 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.642 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.643 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.674846+0000", 2024-04-19T02:03:20.643 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6243999999999997e-05 2024-04-19T02:03:20.643 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.643 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.643 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.643 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.643 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.643 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.2s0 32/26 ECSubReadReply(tid=74, attrs_read=0))", 2024-04-19T02:03:20.643 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.663949+0000", 2024-04-19T02:03:20.643 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 3.903165343, 2024-04-19T02:03:20.643 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.011325732999999999, 2024-04-19T02:03:20.643 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.643 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.643 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:20.643 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.644 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.644 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.644 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.663949+0000", 2024-04-19T02:03:20.644 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.644 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.644 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.644 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.644 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.663949+0000", 2024-04-19T02:03:20.644 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.644 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.644 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.644 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.644 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.663949+0000", 2024-04-19T02:03:20.644 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.645 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.645 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.645 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.645 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.663949+0000", 2024-04-19T02:03:20.645 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.645 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.645 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.645 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.645 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.664082+0000", 2024-04-19T02:03:20.645 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013250600000000001 2024-04-19T02:03:20.645 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.645 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.645 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.645 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.664122+0000", 2024-04-19T02:03:20.645 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0055999999999997e-05 2024-04-19T02:03:20.645 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.646 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.646 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.646 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.674882+0000", 2024-04-19T02:03:20.646 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010760525 2024-04-19T02:03:20.646 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.646 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.646 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.646 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.675275+0000", 2024-04-19T02:03:20.646 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00039264600000000002 2024-04-19T02:03:20.646 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.646 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.646 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.646 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.646 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.647 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.2s0 32/26 ECSubReadReply(tid=75, attrs_read=0))", 2024-04-19T02:03:20.647 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.667310+0000", 2024-04-19T02:03:20.647 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 3.8998051390000001, 2024-04-19T02:03:20.647 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0086427229999999997, 2024-04-19T02:03:20.647 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.647 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.647 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:20.647 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.647 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.647 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.647 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.667310+0000", 2024-04-19T02:03:20.647 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.647 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.648 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.648 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.648 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.667310+0000", 2024-04-19T02:03:20.648 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.648 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.648 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.648 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.648 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.667310+0000", 2024-04-19T02:03:20.648 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.648 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.648 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.648 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.648 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.667310+0000", 2024-04-19T02:03:20.648 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.649 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.649 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.649 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.649 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.667358+0000", 2024-04-19T02:03:20.649 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.8041000000000003e-05 2024-04-19T02:03:20.649 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.649 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.649 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.649 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.667374+0000", 2024-04-19T02:03:20.649 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.658e-05 2024-04-19T02:03:20.649 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.649 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.649 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.649 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.675516+0000", 2024-04-19T02:03:20.650 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0081422189999999992 2024-04-19T02:03:20.650 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.650 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.650 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.650 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.675952+0000", 2024-04-19T02:03:20.650 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00043588299999999997 2024-04-19T02:03:20.650 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.650 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.650 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.650 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.650 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.650 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es6 35/26 ECSubWrite(tid=23, reqid=osd.7.0:23, at_version=35'18, trim_to=0'0, roll_forward_to=34'16))", 2024-04-19T02:03:20.650 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.677012+0000", 2024-04-19T02:03:20.650 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.89010251600000001, 2024-04-19T02:03:20.651 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.005089423, 2024-04-19T02:03:20.651 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:20.651 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:20.651 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:20.651 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:20.651 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.651 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:20.651 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.677012+0000", 2024-04-19T02:03:20.651 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.651 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.651 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.651 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:20.651 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.677012+0000", 2024-04-19T02:03:20.651 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:20.652 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.652 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.652 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:20.652 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.677013+0000", 2024-04-19T02:03:20.652 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.37e-07 2024-04-19T02:03:20.652 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.652 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.652 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:20.652 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.677038+0000", 2024-04-19T02:03:20.652 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5347e-05 2024-04-19T02:03:20.652 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.652 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.652 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:20.652 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.677052+0000", 2024-04-19T02:03:20.653 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3386e-05 2024-04-19T02:03:20.653 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.653 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.653 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:20.653 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.677061+0000", 2024-04-19T02:03:20.653 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.448e-06 2024-04-19T02:03:20.653 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.653 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.653 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:20.653 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.677168+0000", 2024-04-19T02:03:20.653 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010681200000000001 2024-04-19T02:03:20.653 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.653 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.653 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:20.654 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.677179+0000", 2024-04-19T02:03:20.654 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0981999999999999e-05 2024-04-19T02:03:20.654 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.654 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.654 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:20.654 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.682060+0000", 2024-04-19T02:03:20.654 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0048808310000000004 2024-04-19T02:03:20.654 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:20.654 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:20.654 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:20.654 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.682102+0000", 2024-04-19T02:03:20.654 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1879999999999999e-05 2024-04-19T02:03:20.654 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.654 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.655 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.655 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:20.655 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:20.655 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:20.656 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-19T02:03:20.657 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:03:20.655+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:03:20.674 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is set 2024-04-19T02:03:20.675 INFO:teuthology.orchestra.run.smithi059.stderr:set pool 3 pgp_num to 16 2024-04-19T02:03:20.682 INFO:tasks.rados.rados.0.smithi059.stdout:111: done (1 left) 2024-04-19T02:03:20.682 INFO:tasks.rados.rados.0.smithi059.stdout:113: done (0 left) 2024-04-19T02:03:20.682 INFO:tasks.rados.rados.0.smithi059.stdout:114: snap_create 2024-04-19T02:03:20.758 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:20.759+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:20.785 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:20.785 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:20.785 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 0 2024-04-19T02:03:20.785 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:20.799 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-19T02:03:20.859 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:20.859+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:20.921 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:20.921 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:20.921 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-04-19T02:03:20.921 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-04-19T02:03:20.921 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:20.935 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-19T02:03:20.962 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:20.963+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:21.059 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:21.059 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-04-19T02:03:21.059 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-04-19T02:03:21.059 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-04-19T02:03:21.059 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.059 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.2s3 31/26 ECSubWrite(tid=65, reqid=client.4509.0:230, at_version=31'24, trim_to=0'0, roll_forward_to=30'22))", 2024-04-19T02:03:21.059 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:15.641337+0000", 2024-04-19T02:03:21.060 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 5.419703084, 2024-04-19T02:03:21.060 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001643455, 2024-04-19T02:03:21.060 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.060 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.060 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:21.060 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.060 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.060 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.060 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:15.641337+0000", 2024-04-19T02:03:21.060 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.060 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.060 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.060 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.060 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:15.641337+0000", 2024-04-19T02:03:21.060 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.060 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.061 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.061 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.061 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:15.641337+0000", 2024-04-19T02:03:21.061 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2599999999999999e-07 2024-04-19T02:03:21.061 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.061 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.061 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.061 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:15.641358+0000", 2024-04-19T02:03:21.061 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1384999999999999e-05 2024-04-19T02:03:21.061 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.061 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.061 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.061 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:15.641370+0000", 2024-04-19T02:03:21.062 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2011000000000001e-05 2024-04-19T02:03:21.062 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.062 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.062 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.062 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:15.641378+0000", 2024-04-19T02:03:21.063 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6799999999999993e-06 2024-04-19T02:03:21.063 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.063 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.063 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.063 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:15.641436+0000", 2024-04-19T02:03:21.063 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7858000000000001e-05 2024-04-19T02:03:21.063 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.064 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.064 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:21.064 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:15.641443+0000", 2024-04-19T02:03:21.064 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9349999999999997e-06 2024-04-19T02:03:21.064 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.064 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.064 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:21.064 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:15.642907+0000", 2024-04-19T02:03:21.065 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0014642979999999999 2024-04-19T02:03:21.065 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.065 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.065 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.065 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:15.642980+0000", 2024-04-19T02:03:21.065 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3162000000000002e-05 2024-04-19T02:03:21.065 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.065 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.065 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.065 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.065 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.065 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.2s3 31/26 ECSubWrite(tid=67, reqid=client.4509.0:231, at_version=31'26, trim_to=0'0, roll_forward_to=31'24))", 2024-04-19T02:03:21.065 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:15.660895+0000", 2024-04-19T02:03:21.065 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 5.4001443570000003, 2024-04-19T02:03:21.065 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002045472, 2024-04-19T02:03:21.066 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.066 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.066 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:21.066 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.066 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.066 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.066 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:15.660895+0000", 2024-04-19T02:03:21.066 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.066 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.066 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.066 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.066 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:15.660895+0000", 2024-04-19T02:03:21.066 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.066 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.067 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.067 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.067 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:15.660897+0000", 2024-04-19T02:03:21.067 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.623e-06 2024-04-19T02:03:21.067 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.067 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.067 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.067 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:15.661454+0000", 2024-04-19T02:03:21.067 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00055703600000000001 2024-04-19T02:03:21.067 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.068 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.068 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.068 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:15.661484+0000", 2024-04-19T02:03:21.068 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0014000000000001e-05 2024-04-19T02:03:21.068 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.068 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.068 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.068 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:15.661505+0000", 2024-04-19T02:03:21.068 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1406000000000001e-05 2024-04-19T02:03:21.068 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.068 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.068 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.068 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:15.661573+0000", 2024-04-19T02:03:21.068 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7036e-05 2024-04-19T02:03:21.068 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.068 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.069 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:21.069 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:15.661578+0000", 2024-04-19T02:03:21.069 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.203e-06 2024-04-19T02:03:21.069 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.069 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.069 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:21.069 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:15.662892+0000", 2024-04-19T02:03:21.069 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0013140070000000001 2024-04-19T02:03:21.069 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.069 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.069 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.069 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:15.662941+0000", 2024-04-19T02:03:21.069 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9147000000000002e-05 2024-04-19T02:03:21.069 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.069 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.070 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.070 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.070 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.070 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.2s3 32/26 ECSubWrite(tid=72, reqid=client.4509.0:236, at_version=32'29, trim_to=0'0, roll_forward_to=31'28))", 2024-04-19T02:03:21.070 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:16.654509+0000", 2024-04-19T02:03:21.070 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.4065308160000001, 2024-04-19T02:03:21.070 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0021830199999999999, 2024-04-19T02:03:21.070 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.070 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.070 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:21.071 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.071 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.071 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.071 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.654509+0000", 2024-04-19T02:03:21.072 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.072 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.072 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.072 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.072 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.654509+0000", 2024-04-19T02:03:21.072 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.072 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.072 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.072 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.072 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.654510+0000", 2024-04-19T02:03:21.072 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.35e-06 2024-04-19T02:03:21.072 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.072 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.072 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.072 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.654661+0000", 2024-04-19T02:03:21.073 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015107900000000001 2024-04-19T02:03:21.073 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.073 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.073 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.073 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.654691+0000", 2024-04-19T02:03:21.073 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9567e-05 2024-04-19T02:03:21.073 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.073 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.073 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.073 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.654713+0000", 2024-04-19T02:03:21.073 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2473e-05 2024-04-19T02:03:21.073 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.073 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.074 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.074 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.654800+0000", 2024-04-19T02:03:21.074 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.6482000000000003e-05 2024-04-19T02:03:21.074 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.074 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.074 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:21.074 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.654810+0000", 2024-04-19T02:03:21.074 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7170000000000003e-06 2024-04-19T02:03:21.074 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.074 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.074 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:21.074 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.656591+0000", 2024-04-19T02:03:21.075 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0017811260000000001 2024-04-19T02:03:21.075 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.075 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.075 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.075 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.656692+0000", 2024-04-19T02:03:21.075 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000101226 2024-04-19T02:03:21.075 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.075 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.075 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.075 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.075 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.075 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.2s3 32/26 ECSubWrite(tid=76, reqid=client.4509.0:236, at_version=0'0, trim_to=0'0, roll_forward_to=32'29))", 2024-04-19T02:03:21.075 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:16.668681+0000", 2024-04-19T02:03:21.075 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.3923586029999999, 2024-04-19T02:03:21.076 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0024522680000000001, 2024-04-19T02:03:21.076 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.076 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.076 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:21.076 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.076 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.076 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.076 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.668681+0000", 2024-04-19T02:03:21.076 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.076 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.076 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.076 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.076 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.668681+0000", 2024-04-19T02:03:21.076 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.076 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.077 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.077 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.077 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.668682+0000", 2024-04-19T02:03:21.077 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6799999999999999e-07 2024-04-19T02:03:21.077 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.077 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.077 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.077 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.668695+0000", 2024-04-19T02:03:21.077 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3251999999999999e-05 2024-04-19T02:03:21.077 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.078 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.078 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.078 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.668711+0000", 2024-04-19T02:03:21.078 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6028999999999998e-05 2024-04-19T02:03:21.078 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.078 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.078 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.078 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.668719+0000", 2024-04-19T02:03:21.078 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7870000000000002e-06 2024-04-19T02:03:21.078 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.079 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.079 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.079 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.668859+0000", 2024-04-19T02:03:21.079 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013957800000000001 2024-04-19T02:03:21.079 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.079 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.079 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:21.079 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.668887+0000", 2024-04-19T02:03:21.079 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8022999999999999e-05 2024-04-19T02:03:21.079 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.079 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.079 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:21.079 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.671057+0000", 2024-04-19T02:03:21.079 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0021705660000000001 2024-04-19T02:03:21.080 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.080 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.080 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.080 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.671133+0000", 2024-04-19T02:03:21.080 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6265000000000002e-05 2024-04-19T02:03:21.080 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.080 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.080 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.080 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.080 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.080 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.2s3 32/26 ECSubWrite(tid=77, reqid=osd.3.0:77, at_version=32'31, trim_to=0'0, roll_forward_to=32'29))", 2024-04-19T02:03:21.080 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:16.671715+0000", 2024-04-19T02:03:21.080 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.3893243589999997, 2024-04-19T02:03:21.080 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0025057629999999998, 2024-04-19T02:03:21.080 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.081 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.081 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:21.081 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.081 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.081 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.081 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.671715+0000", 2024-04-19T02:03:21.081 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.081 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.081 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.081 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.081 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.671715+0000", 2024-04-19T02:03:21.081 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.081 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.081 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.081 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.081 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.671716+0000", 2024-04-19T02:03:21.082 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4700000000000001e-07 2024-04-19T02:03:21.082 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.082 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.082 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.082 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.671733+0000", 2024-04-19T02:03:21.082 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7354999999999999e-05 2024-04-19T02:03:21.082 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.082 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.082 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.082 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.671745+0000", 2024-04-19T02:03:21.082 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1799e-05 2024-04-19T02:03:21.083 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.083 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.083 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.083 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.671751+0000", 2024-04-19T02:03:21.083 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1530000000000002e-06 2024-04-19T02:03:21.083 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.083 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.083 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.083 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.671856+0000", 2024-04-19T02:03:21.083 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000104267 2024-04-19T02:03:21.083 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.083 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.083 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:21.083 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.671870+0000", 2024-04-19T02:03:21.083 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4657e-05 2024-04-19T02:03:21.083 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.084 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.084 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:21.084 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.674133+0000", 2024-04-19T02:03:21.084 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002262343 2024-04-19T02:03:21.084 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.084 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.084 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.084 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.674221+0000", 2024-04-19T02:03:21.084 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8541999999999994e-05 2024-04-19T02:03:21.084 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.084 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.084 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.084 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.084 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.085 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.2s3 32/26 ECSubWrite(tid=78, reqid=osd.3.0:78, at_version=32'33, trim_to=0'0, roll_forward_to=32'29))", 2024-04-19T02:03:21.085 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:16.673374+0000", 2024-04-19T02:03:21.085 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.3876662270000004, 2024-04-19T02:03:21.085 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003245375, 2024-04-19T02:03:21.085 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.085 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.085 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:21.085 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.085 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.085 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.085 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.673374+0000", 2024-04-19T02:03:21.085 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.085 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.086 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.086 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.086 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.673374+0000", 2024-04-19T02:03:21.086 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.086 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.086 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.086 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.086 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.673374+0000", 2024-04-19T02:03:21.086 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5499999999999998e-07 2024-04-19T02:03:21.086 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.086 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.086 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.086 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.673389+0000", 2024-04-19T02:03:21.086 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5056e-05 2024-04-19T02:03:21.087 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.087 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.087 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.087 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.673400+0000", 2024-04-19T02:03:21.087 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0499999999999999e-05 2024-04-19T02:03:21.087 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.087 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.087 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.087 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.673405+0000", 2024-04-19T02:03:21.087 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5389999999999998e-06 2024-04-19T02:03:21.087 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.087 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.087 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.087 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.673506+0000", 2024-04-19T02:03:21.087 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010030300000000001 2024-04-19T02:03:21.088 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.088 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.088 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:21.088 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.673528+0000", 2024-04-19T02:03:21.088 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2395000000000002e-05 2024-04-19T02:03:21.088 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.088 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.088 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:21.089 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.676574+0000", 2024-04-19T02:03:21.089 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0030464699999999999 2024-04-19T02:03:21.089 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.089 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.089 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.089 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:16.676619+0000", 2024-04-19T02:03:21.089 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4557e-05 2024-04-19T02:03:21.089 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.089 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.090 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.090 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.090 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.090 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.es1 35/26 ECSubWrite(tid=23, reqid=osd.7.0:23, at_version=35'18, trim_to=0'0, roll_forward_to=34'16))", 2024-04-19T02:03:21.090 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:19.676794+0000", 2024-04-19T02:03:21.090 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.3842457210000001, 2024-04-19T02:03:21.090 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012162914, 2024-04-19T02:03:21.090 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.090 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.091 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:21.091 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.091 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.091 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.091 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.676794+0000", 2024-04-19T02:03:21.091 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.091 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.091 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.091 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.092 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.676794+0000", 2024-04-19T02:03:21.092 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.092 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.092 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.092 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.092 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.676795+0000", 2024-04-19T02:03:21.092 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.04e-06 2024-04-19T02:03:21.093 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.093 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.093 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.093 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.677097+0000", 2024-04-19T02:03:21.093 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00030220099999999999 2024-04-19T02:03:21.093 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.093 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.094 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.094 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.677137+0000", 2024-04-19T02:03:21.094 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9756999999999999e-05 2024-04-19T02:03:21.094 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.094 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.095 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.095 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.677442+0000", 2024-04-19T02:03:21.095 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00030463799999999998 2024-04-19T02:03:21.095 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.095 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.095 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.095 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.677615+0000", 2024-04-19T02:03:21.095 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017371899999999999 2024-04-19T02:03:21.096 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.096 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.096 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:21.096 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.677641+0000", 2024-04-19T02:03:21.096 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5908e-05 2024-04-19T02:03:21.096 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.096 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.096 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:21.097 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.688872+0000", 2024-04-19T02:03:21.097 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011230957999999999 2024-04-19T02:03:21.097 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.097 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.097 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.097 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.688957+0000", 2024-04-19T02:03:21.097 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4692999999999997e-05 2024-04-19T02:03:21.098 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.098 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.098 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.098 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.098 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.098 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.fs1 35/26 ECSubWrite(tid=54, reqid=client.4509.0:271, at_version=35'14, trim_to=0'0, roll_forward_to=34'13))", 2024-04-19T02:03:21.098 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:19.685169+0000", 2024-04-19T02:03:21.098 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.375870707, 2024-04-19T02:03:21.099 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0060707540000000003, 2024-04-19T02:03:21.099 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.099 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.099 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:21.099 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.099 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.099 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.099 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.685169+0000", 2024-04-19T02:03:21.099 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.100 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.100 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.100 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.100 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.685169+0000", 2024-04-19T02:03:21.100 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.100 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.100 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.100 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.101 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.685170+0000", 2024-04-19T02:03:21.101 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7300000000000005e-07 2024-04-19T02:03:21.101 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.101 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.101 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.101 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.685600+0000", 2024-04-19T02:03:21.101 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00042986100000000002 2024-04-19T02:03:21.101 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.101 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.102 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.102 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.685633+0000", 2024-04-19T02:03:21.102 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3147e-05 2024-04-19T02:03:21.102 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.102 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.102 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.102 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.685656+0000", 2024-04-19T02:03:21.103 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2798000000000001e-05 2024-04-19T02:03:21.103 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.103 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.103 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.103 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.686185+0000", 2024-04-19T02:03:21.103 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00052935700000000005 2024-04-19T02:03:21.103 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.104 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.104 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:21.104 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.686209+0000", 2024-04-19T02:03:21.104 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4324e-05 2024-04-19T02:03:21.104 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.104 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.104 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:21.104 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.691159+0000", 2024-04-19T02:03:21.104 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0049495809999999998 2024-04-19T02:03:21.104 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.105 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.105 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.105 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.691240+0000", 2024-04-19T02:03:21.105 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0913000000000003e-05 2024-04-19T02:03:21.105 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.105 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.105 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.105 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.106 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.106 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.2s3 35/26 ECSubWrite(tid=95, reqid=osd.3.0:92, at_version=0'0, trim_to=0'0, roll_forward_to=35'46))", 2024-04-19T02:03:21.106 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:19.685416+0000", 2024-04-19T02:03:21.106 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.375623858, 2024-04-19T02:03:21.106 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0058981260000000001, 2024-04-19T02:03:21.106 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.106 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.106 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:21.106 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.107 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.107 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.107 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.685416+0000", 2024-04-19T02:03:21.107 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.107 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.107 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.107 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.107 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.685416+0000", 2024-04-19T02:03:21.107 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.108 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.108 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.108 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.108 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.685417+0000", 2024-04-19T02:03:21.108 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.327e-06 2024-04-19T02:03:21.108 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.108 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.108 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.109 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.685444+0000", 2024-04-19T02:03:21.109 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7197999999999999e-05 2024-04-19T02:03:21.109 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.109 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.109 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.109 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.685471+0000", 2024-04-19T02:03:21.109 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6497000000000001e-05 2024-04-19T02:03:21.109 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.109 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.109 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.110 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.685493+0000", 2024-04-19T02:03:21.110 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1948999999999999e-05 2024-04-19T02:03:21.110 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.110 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.110 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.110 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.685643+0000", 2024-04-19T02:03:21.110 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015002400000000001 2024-04-19T02:03:21.110 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.111 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.111 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:21.111 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.685667+0000", 2024-04-19T02:03:21.111 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4083999999999998e-05 2024-04-19T02:03:21.111 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.111 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.111 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:21.112 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.690989+0000", 2024-04-19T02:03:21.112 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0053224170000000003 2024-04-19T02:03:21.112 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.112 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.112 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.112 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.691314+0000", 2024-04-19T02:03:21.112 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00032463000000000001 2024-04-19T02:03:21.112 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.112 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.112 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.112 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.112 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.112 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.fs1 35/26 ECSubWrite(tid=55, reqid=client.4509.0:272, at_version=35'15, trim_to=0'0, roll_forward_to=34'13))", 2024-04-19T02:03:21.112 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:19.689250+0000", 2024-04-19T02:03:21.113 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.3717893480000001, 2024-04-19T02:03:21.113 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0064390369999999999, 2024-04-19T02:03:21.113 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.113 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.113 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:21.113 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.113 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.113 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.113 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.689250+0000", 2024-04-19T02:03:21.113 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.113 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.113 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.113 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.113 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.689250+0000", 2024-04-19T02:03:21.113 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.113 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.114 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.114 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.114 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.689252+0000", 2024-04-19T02:03:21.114 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.75e-06 2024-04-19T02:03:21.114 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.114 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.114 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.114 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.689599+0000", 2024-04-19T02:03:21.114 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000346664 2024-04-19T02:03:21.114 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.114 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.114 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.114 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.689625+0000", 2024-04-19T02:03:21.115 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6274999999999998e-05 2024-04-19T02:03:21.115 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.115 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.115 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.115 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.689642+0000", 2024-04-19T02:03:21.115 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7278999999999998e-05 2024-04-19T02:03:21.115 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.115 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.115 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.115 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.689784+0000", 2024-04-19T02:03:21.115 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014182100000000001 2024-04-19T02:03:21.115 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.115 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.115 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:21.115 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.689809+0000", 2024-04-19T02:03:21.116 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4335e-05 2024-04-19T02:03:21.116 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.116 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.116 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:21.116 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.695594+0000", 2024-04-19T02:03:21.116 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0057853239999999997 2024-04-19T02:03:21.116 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.116 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.116 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.116 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.695689+0000", 2024-04-19T02:03:21.116 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5588999999999996e-05 2024-04-19T02:03:21.116 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.116 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.116 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.116 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.117 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.117 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.3s4 35/26 ECSubWrite(tid=30, reqid=osd.10.0:29, at_version=0'0, trim_to=0'0, roll_forward_to=35'8))", 2024-04-19T02:03:21.117 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:19.691597+0000", 2024-04-19T02:03:21.117 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.369442636, 2024-04-19T02:03:21.117 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0066070699999999996, 2024-04-19T02:03:21.117 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.117 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.117 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:21.117 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.117 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.117 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.117 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.691597+0000", 2024-04-19T02:03:21.117 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.117 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.117 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.118 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.118 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.691597+0000", 2024-04-19T02:03:21.118 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.118 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.118 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.118 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.118 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.691599+0000", 2024-04-19T02:03:21.118 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.057e-06 2024-04-19T02:03:21.118 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.118 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.118 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.118 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.691627+0000", 2024-04-19T02:03:21.118 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7373999999999999e-05 2024-04-19T02:03:21.118 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.119 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.119 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.119 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.691661+0000", 2024-04-19T02:03:21.119 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.489e-05 2024-04-19T02:03:21.119 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.119 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.119 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.119 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.691683+0000", 2024-04-19T02:03:21.119 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1285999999999999e-05 2024-04-19T02:03:21.119 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.119 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.119 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.119 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.691833+0000", 2024-04-19T02:03:21.119 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000149977 2024-04-19T02:03:21.119 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.119 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.120 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:21.120 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.691856+0000", 2024-04-19T02:03:21.120 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3640999999999999e-05 2024-04-19T02:03:21.120 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.120 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.120 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:21.120 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.698113+0000", 2024-04-19T02:03:21.120 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.006256246 2024-04-19T02:03:21.120 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.120 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.120 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.120 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.698204+0000", 2024-04-19T02:03:21.120 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1599000000000002e-05 2024-04-19T02:03:21.120 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.120 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.121 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.121 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.121 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.121 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 35/26 ECSubWriteReply(tid=60, last_complete=35'24, committed=1, applied=1))", 2024-04-19T02:03:21.121 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:19.694222+0000", 2024-04-19T02:03:21.121 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.3668179, 2024-04-19T02:03:21.121 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0015688099999999999, 2024-04-19T02:03:21.121 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.121 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.121 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:21.121 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.121 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.121 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.121 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.694222+0000", 2024-04-19T02:03:21.121 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.122 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.122 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.122 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.122 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.694222+0000", 2024-04-19T02:03:21.122 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.122 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.122 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.122 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.122 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.694224+0000", 2024-04-19T02:03:21.122 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7400000000000001e-06 2024-04-19T02:03:21.122 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.122 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.122 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.122 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.694236+0000", 2024-04-19T02:03:21.122 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2159e-05 2024-04-19T02:03:21.123 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.123 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.123 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.123 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.694267+0000", 2024-04-19T02:03:21.123 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0867000000000001e-05 2024-04-19T02:03:21.123 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.123 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.123 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.123 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.694286+0000", 2024-04-19T02:03:21.123 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9361999999999999e-05 2024-04-19T02:03:21.123 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.123 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.123 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.123 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.695756+0000", 2024-04-19T02:03:21.123 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001469665 2024-04-19T02:03:21.123 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.124 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.124 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.124 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.695791+0000", 2024-04-19T02:03:21.124 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5017e-05 2024-04-19T02:03:21.124 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.124 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.124 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.124 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.124 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.124 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 35/26 ECSubWriteReply(tid=60, last_complete=35'24, committed=1, applied=1))", 2024-04-19T02:03:21.124 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:19.694607+0000", 2024-04-19T02:03:21.124 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.366432662, 2024-04-19T02:03:21.124 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0012592689999999999, 2024-04-19T02:03:21.124 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.124 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.124 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:21.125 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.125 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.125 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.125 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.694607+0000", 2024-04-19T02:03:21.125 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.125 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.125 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.125 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.125 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.694607+0000", 2024-04-19T02:03:21.125 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.125 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.125 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.125 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.125 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.694609+0000", 2024-04-19T02:03:21.126 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0200000000000001e-06 2024-04-19T02:03:21.126 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.126 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.126 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.126 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.694622+0000", 2024-04-19T02:03:21.126 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2758999999999999e-05 2024-04-19T02:03:21.126 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.126 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.126 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.126 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.694652+0000", 2024-04-19T02:03:21.126 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0193999999999999e-05 2024-04-19T02:03:21.126 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.126 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.126 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.127 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.694950+0000", 2024-04-19T02:03:21.127 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00029758900000000001 2024-04-19T02:03:21.127 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.127 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.127 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.127 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.695829+0000", 2024-04-19T02:03:21.127 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00087924699999999999 2024-04-19T02:03:21.127 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.127 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.127 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.127 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.695866+0000", 2024-04-19T02:03:21.127 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7459999999999997e-05 2024-04-19T02:03:21.127 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.127 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.127 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.128 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.128 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.128 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.fs1 35/26 ECSubWrite(tid=56, reqid=client.4509.0:273, at_version=35'16, trim_to=0'0, roll_forward_to=34'13))", 2024-04-19T02:03:21.128 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:19.695692+0000", 2024-04-19T02:03:21.128 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.365348198, 2024-04-19T02:03:21.128 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0045888869999999998, 2024-04-19T02:03:21.128 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.128 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.128 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:21.128 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.128 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.128 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.128 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.695692+0000", 2024-04-19T02:03:21.128 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.128 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.129 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.129 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.129 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.695692+0000", 2024-04-19T02:03:21.129 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.129 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.129 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.129 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.129 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.695693+0000", 2024-04-19T02:03:21.129 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8560000000000001e-06 2024-04-19T02:03:21.129 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.129 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.129 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.129 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.695733+0000", 2024-04-19T02:03:21.129 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9959999999999997e-05 2024-04-19T02:03:21.129 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.129 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.130 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.130 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.695763+0000", 2024-04-19T02:03:21.130 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9111999999999998e-05 2024-04-19T02:03:21.130 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.130 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.130 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.130 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.695785+0000", 2024-04-19T02:03:21.130 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2147999999999999e-05 2024-04-19T02:03:21.130 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.130 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.130 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.130 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.695954+0000", 2024-04-19T02:03:21.130 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016949900000000001 2024-04-19T02:03:21.130 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.131 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.131 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:21.131 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.695978+0000", 2024-04-19T02:03:21.131 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4051000000000001e-05 2024-04-19T02:03:21.131 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.131 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.131 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:21.131 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.700184+0000", 2024-04-19T02:03:21.131 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0042055260000000002 2024-04-19T02:03:21.131 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.131 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.131 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.131 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.700280+0000", 2024-04-19T02:03:21.131 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6735000000000001e-05 2024-04-19T02:03:21.131 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.131 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.131 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.132 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.132 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.132 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.es1 35/26 ECSubWrite(tid=24, reqid=osd.7.0:23, at_version=0'0, trim_to=0'0, roll_forward_to=35'18))", 2024-04-19T02:03:21.132 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:19.697012+0000", 2024-04-19T02:03:21.132 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.3640280040000001, 2024-04-19T02:03:21.132 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0060535800000000002, 2024-04-19T02:03:21.132 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.132 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.132 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:21.132 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.132 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.132 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.133 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.697012+0000", 2024-04-19T02:03:21.133 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.133 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.133 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.133 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.133 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.697012+0000", 2024-04-19T02:03:21.133 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.133 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.133 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.133 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.133 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.697013+0000", 2024-04-19T02:03:21.133 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.66e-06 2024-04-19T02:03:21.133 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.133 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.133 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.133 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.697038+0000", 2024-04-19T02:03:21.134 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4091e-05 2024-04-19T02:03:21.134 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.134 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.134 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.134 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.697075+0000", 2024-04-19T02:03:21.134 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7307e-05 2024-04-19T02:03:21.134 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.134 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.134 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.134 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.697097+0000", 2024-04-19T02:03:21.134 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2311999999999999e-05 2024-04-19T02:03:21.135 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.135 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.135 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.135 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.697276+0000", 2024-04-19T02:03:21.135 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017881200000000001 2024-04-19T02:03:21.135 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.135 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.135 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:21.135 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.697303+0000", 2024-04-19T02:03:21.135 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7041000000000001e-05 2024-04-19T02:03:21.135 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.136 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.136 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:21.136 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.702976+0000", 2024-04-19T02:03:21.136 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0056733039999999997 2024-04-19T02:03:21.136 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.136 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.136 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.136 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.703065+0000", 2024-04-19T02:03:21.136 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.9053000000000003e-05 2024-04-19T02:03:21.136 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.136 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.136 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.136 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.136 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.137 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 35/26 ECSubWriteReply(tid=61, last_complete=35'26, committed=1, applied=1))", 2024-04-19T02:03:21.137 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:19.700845+0000", 2024-04-19T02:03:21.137 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.360194911, 2024-04-19T02:03:21.137 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0031391240000000001, 2024-04-19T02:03:21.137 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.137 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.137 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:21.137 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.137 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.137 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.137 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.700845+0000", 2024-04-19T02:03:21.137 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.137 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.137 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.137 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.137 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.700845+0000", 2024-04-19T02:03:21.138 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.138 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.138 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.138 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.138 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.700846+0000", 2024-04-19T02:03:21.138 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9699999999999995e-07 2024-04-19T02:03:21.138 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.138 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.138 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.138 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.700852+0000", 2024-04-19T02:03:21.138 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.353e-06 2024-04-19T02:03:21.138 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.138 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.138 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.139 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.700873+0000", 2024-04-19T02:03:21.139 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0927000000000001e-05 2024-04-19T02:03:21.139 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.139 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.139 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.139 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.700886+0000", 2024-04-19T02:03:21.139 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3056e-05 2024-04-19T02:03:21.139 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.139 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.139 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.139 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.700989+0000", 2024-04-19T02:03:21.139 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010317199999999999 2024-04-19T02:03:21.139 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.139 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.139 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.139 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.703984+0000", 2024-04-19T02:03:21.140 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0029948190000000001 2024-04-19T02:03:21.140 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.140 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.140 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.140 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.140 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.140 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.fs1 35/26 ECSubWrite(tid=57, reqid=client.4509.0:273, at_version=0'0, trim_to=0'0, roll_forward_to=35'16))", 2024-04-19T02:03:21.140 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:19.701731+0000", 2024-04-19T02:03:21.140 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.3593089540000001, 2024-04-19T02:03:21.140 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0026743880000000002, 2024-04-19T02:03:21.140 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.140 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.140 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:21.140 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.140 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.140 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.141 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.701731+0000", 2024-04-19T02:03:21.141 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.141 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.141 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.141 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.141 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.701731+0000", 2024-04-19T02:03:21.141 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.141 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.141 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.141 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.141 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.701732+0000", 2024-04-19T02:03:21.141 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6729999999999999e-06 2024-04-19T02:03:21.141 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.141 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.141 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.141 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.701776+0000", 2024-04-19T02:03:21.142 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3373e-05 2024-04-19T02:03:21.142 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.142 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.142 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.142 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.701808+0000", 2024-04-19T02:03:21.142 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.2437999999999997e-05 2024-04-19T02:03:21.142 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.142 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.142 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.142 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.701834+0000", 2024-04-19T02:03:21.142 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5401e-05 2024-04-19T02:03:21.142 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.142 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.142 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.143 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.702175+0000", 2024-04-19T02:03:21.143 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000340818 2024-04-19T02:03:21.143 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.143 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.143 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:21.143 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.702201+0000", 2024-04-19T02:03:21.143 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6480999999999999e-05 2024-04-19T02:03:21.143 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.143 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.143 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:21.143 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.704325+0000", 2024-04-19T02:03:21.143 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0021240880000000001 2024-04-19T02:03:21.143 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.143 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.143 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.144 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.704405+0000", 2024-04-19T02:03:21.144 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0116000000000003e-05 2024-04-19T02:03:21.144 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.144 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.144 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.144 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.144 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.144 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 35/26 ECSubWriteReply(tid=62, last_complete=35'26, committed=1, applied=1))", 2024-04-19T02:03:21.144 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:19.703421+0000", 2024-04-19T02:03:21.144 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.357618397, 2024-04-19T02:03:21.144 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001461546, 2024-04-19T02:03:21.144 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.145 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.145 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:21.145 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.145 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.145 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.145 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.703421+0000", 2024-04-19T02:03:21.145 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.145 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.145 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.145 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.145 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.703421+0000", 2024-04-19T02:03:21.145 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.145 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.145 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.145 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.146 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.703423+0000", 2024-04-19T02:03:21.146 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.55e-06 2024-04-19T02:03:21.146 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.146 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.146 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.146 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.703435+0000", 2024-04-19T02:03:21.146 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2221999999999999e-05 2024-04-19T02:03:21.146 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.146 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.146 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.146 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.703465+0000", 2024-04-19T02:03:21.146 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0008000000000001e-05 2024-04-19T02:03:21.146 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.147 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.147 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.147 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.703487+0000", 2024-04-19T02:03:21.147 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1424e-05 2024-04-19T02:03:21.147 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.147 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.147 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.147 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.704835+0000", 2024-04-19T02:03:21.147 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001348269 2024-04-19T02:03:21.148 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.148 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.148 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.148 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.704883+0000", 2024-04-19T02:03:21.148 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8072999999999999e-05 2024-04-19T02:03:21.148 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.148 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.149 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.149 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.149 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.149 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.3s4 35/26 ECSubWrite(tid=36, reqid=client.4509.0:289, at_version=35'9, trim_to=0'0, roll_forward_to=35'8))", 2024-04-19T02:03:21.149 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:19.786614+0000", 2024-04-19T02:03:21.149 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.2744253379999999, 2024-04-19T02:03:21.149 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003034237, 2024-04-19T02:03:21.150 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.150 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.150 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:21.150 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.150 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.150 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.151 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.786614+0000", 2024-04-19T02:03:21.151 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.151 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.151 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.151 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.151 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.786614+0000", 2024-04-19T02:03:21.152 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.152 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.152 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.152 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.152 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.786616+0000", 2024-04-19T02:03:21.152 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.454e-06 2024-04-19T02:03:21.152 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.153 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.153 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.153 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.786914+0000", 2024-04-19T02:03:21.153 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000298067 2024-04-19T02:03:21.153 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.153 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.153 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.154 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.786941+0000", 2024-04-19T02:03:21.154 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7120000000000001e-05 2024-04-19T02:03:21.154 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.154 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.154 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.154 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.786961+0000", 2024-04-19T02:03:21.155 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0075999999999999e-05 2024-04-19T02:03:21.155 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.155 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.155 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.155 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.787092+0000", 2024-04-19T02:03:21.155 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013094800000000001 2024-04-19T02:03:21.155 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.156 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.156 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:21.156 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.787105+0000", 2024-04-19T02:03:21.156 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2833999999999999e-05 2024-04-19T02:03:21.156 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.156 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.157 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:21.157 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.789533+0000", 2024-04-19T02:03:21.157 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002427935 2024-04-19T02:03:21.157 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.157 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.157 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.158 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.789649+0000", 2024-04-19T02:03:21.158 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000115803 2024-04-19T02:03:21.158 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.158 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.158 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.158 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.158 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.159 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.fs1 35/26 ECSubWrite(tid=65, reqid=client.4509.0:290, at_version=0'0, trim_to=0'0, roll_forward_to=35'17))", 2024-04-19T02:03:21.159 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:19.788269+0000", 2024-04-19T02:03:21.159 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.2727708790000001, 2024-04-19T02:03:21.159 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0021961369999999999, 2024-04-19T02:03:21.159 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.159 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.159 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:21.160 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.160 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.160 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.160 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.788269+0000", 2024-04-19T02:03:21.160 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.160 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.160 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.161 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.161 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.788269+0000", 2024-04-19T02:03:21.161 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.161 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.161 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.161 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.162 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.788275+0000", 2024-04-19T02:03:21.162 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.297e-06 2024-04-19T02:03:21.162 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.162 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.162 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.162 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.788293+0000", 2024-04-19T02:03:21.162 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7617000000000001e-05 2024-04-19T02:03:21.163 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.163 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.163 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.163 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.788312+0000", 2024-04-19T02:03:21.163 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9669000000000001e-05 2024-04-19T02:03:21.163 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.164 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.164 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.164 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.788325+0000", 2024-04-19T02:03:21.164 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2403e-05 2024-04-19T02:03:21.164 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.164 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.165 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.165 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.788440+0000", 2024-04-19T02:03:21.165 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000115566 2024-04-19T02:03:21.165 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.165 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.165 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:21.165 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.788451+0000", 2024-04-19T02:03:21.166 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0641e-05 2024-04-19T02:03:21.166 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.166 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.166 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:21.166 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.790394+0000", 2024-04-19T02:03:21.166 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001943169 2024-04-19T02:03:21.167 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.167 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.167 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.167 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.790465+0000", 2024-04-19T02:03:21.167 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0774999999999999e-05 2024-04-19T02:03:21.167 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.168 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.168 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.168 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.168 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.168 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:21.171 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-19T02:03:21.171 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:21.171+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:21.271 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:21.271+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:21.339 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:21.340 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:21.340 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 0 2024-04-19T02:03:21.340 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:21.353 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-19T02:03:21.372 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:21.371+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:21.473 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:21.475+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:21.515 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:21.515 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:21.515 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-04-19T02:03:21.515 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-04-19T02:03:21.515 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:21.530 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-19T02:03:21.574 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:21.575+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:21.656 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:21.656 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-04-19T02:03:21.656 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-04-19T02:03:21.656 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-04-19T02:03:21.656 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.657 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:21 1.0 e19/18 1:30306672:devicehealth::main.db.0000000000000000:head v 19'11, mlcod=19'8)", 2024-04-19T02:03:21.657 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:00.012811+0000", 2024-04-19T02:03:21.657 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 21.644104128999999, 2024-04-19T02:03:21.657 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0053931220000000002, 2024-04-19T02:03:21.657 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.657 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.657 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:21.657 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.657 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.658 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.658 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.012811+0000", 2024-04-19T02:03:21.658 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.658 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.658 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.658 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.658 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.012811+0000", 2024-04-19T02:03:21.658 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.659 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.659 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.659 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.659 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.012812+0000", 2024-04-19T02:03:21.659 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7899999999999998e-07 2024-04-19T02:03:21.659 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.659 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.659 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.659 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.012925+0000", 2024-04-19T02:03:21.660 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011290699999999999 2024-04-19T02:03:21.660 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.660 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.660 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.660 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.012934+0000", 2024-04-19T02:03:21.660 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.611e-06 2024-04-19T02:03:21.660 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.660 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.660 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.661 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.012940+0000", 2024-04-19T02:03:21.661 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4399999999999996e-06 2024-04-19T02:03:21.661 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.661 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.661 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.661 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.013020+0000", 2024-04-19T02:03:21.661 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.9740999999999994e-05 2024-04-19T02:03:21.661 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.662 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.662 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:21.662 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.013056+0000", 2024-04-19T02:03:21.662 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6247e-05 2024-04-19T02:03:21.662 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.662 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.662 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:21.662 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.018113+0000", 2024-04-19T02:03:21.662 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0050570179999999999 2024-04-19T02:03:21.663 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.663 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.663 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.663 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.018204+0000", 2024-04-19T02:03:21.663 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1478999999999996e-05 2024-04-19T02:03:21.663 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.663 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.663 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.663 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.664 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.664 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:22 1.0 e19/18 1:30306672:devicehealth::main.db.0000000000000000:head v 19'12, mlcod=19'8)", 2024-04-19T02:03:21.664 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:00.014091+0000", 2024-04-19T02:03:21.664 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 21.642824008000002, 2024-04-19T02:03:21.664 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0047229300000000002, 2024-04-19T02:03:21.664 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.664 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.664 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:21.665 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.665 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.665 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.665 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.014091+0000", 2024-04-19T02:03:21.665 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.665 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.665 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.665 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.665 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.014091+0000", 2024-04-19T02:03:21.666 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.666 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.666 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.666 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.666 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.014092+0000", 2024-04-19T02:03:21.666 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.75e-07 2024-04-19T02:03:21.666 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.666 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.666 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.667 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.014097+0000", 2024-04-19T02:03:21.667 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7709999999999997e-06 2024-04-19T02:03:21.667 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.667 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.667 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.667 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.014106+0000", 2024-04-19T02:03:21.667 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7210000000000006e-06 2024-04-19T02:03:21.667 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.667 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.668 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.668 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.014112+0000", 2024-04-19T02:03:21.668 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.22e-06 2024-04-19T02:03:21.668 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.668 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.668 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.668 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.014174+0000", 2024-04-19T02:03:21.668 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2843999999999997e-05 2024-04-19T02:03:21.668 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.669 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.669 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:21.669 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.014215+0000", 2024-04-19T02:03:21.669 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0083999999999998e-05 2024-04-19T02:03:21.669 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.669 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.669 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:21.669 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.018755+0000", 2024-04-19T02:03:21.669 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0045405619999999997 2024-04-19T02:03:21.669 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.670 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.670 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.670 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.018814+0000", 2024-04-19T02:03:21.670 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9052999999999999e-05 2024-04-19T02:03:21.670 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.670 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.670 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.670 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.670 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.670 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:84 1.0 e19/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'54, mlcod=19'50)", 2024-04-19T02:03:21.670 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:00.136060+0000", 2024-04-19T02:03:21.670 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 21.520855291, 2024-04-19T02:03:21.671 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0051945680000000001, 2024-04-19T02:03:21.671 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.671 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.671 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:21.671 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.671 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.671 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.671 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.136060+0000", 2024-04-19T02:03:21.671 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.671 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.671 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.672 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.672 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.136060+0000", 2024-04-19T02:03:21.672 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.672 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.672 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.672 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.672 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.136062+0000", 2024-04-19T02:03:21.672 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7600000000000001e-06 2024-04-19T02:03:21.672 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.672 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.672 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.672 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.136120+0000", 2024-04-19T02:03:21.672 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7834000000000002e-05 2024-04-19T02:03:21.672 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.672 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.673 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.673 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.136148+0000", 2024-04-19T02:03:21.673 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8062999999999999e-05 2024-04-19T02:03:21.673 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.673 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.673 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.673 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.136161+0000", 2024-04-19T02:03:21.673 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3033e-05 2024-04-19T02:03:21.673 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.673 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.673 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.673 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.136252+0000", 2024-04-19T02:03:21.673 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1552999999999996e-05 2024-04-19T02:03:21.673 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.673 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.674 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:21.674 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.136284+0000", 2024-04-19T02:03:21.674 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1893000000000001e-05 2024-04-19T02:03:21.674 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.674 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.674 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:21.674 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.141186+0000", 2024-04-19T02:03:21.674 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0049016119999999996 2024-04-19T02:03:21.674 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.674 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.674 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.674 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.141255+0000", 2024-04-19T02:03:21.674 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8819999999999995e-05 2024-04-19T02:03:21.674 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.674 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.675 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.675 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.675 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.675 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:85 1.0 e19/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'55, mlcod=19'51)", 2024-04-19T02:03:21.675 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:00.137394+0000", 2024-04-19T02:03:21.675 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 21.519521656999999, 2024-04-19T02:03:21.675 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0064003139999999998, 2024-04-19T02:03:21.675 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.675 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.675 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:21.675 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.675 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.675 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.675 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.137394+0000", 2024-04-19T02:03:21.676 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.676 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.676 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.676 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.676 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.137394+0000", 2024-04-19T02:03:21.676 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.676 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.676 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.676 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.676 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.137395+0000", 2024-04-19T02:03:21.676 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.156e-06 2024-04-19T02:03:21.676 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.676 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.676 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.676 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.137403+0000", 2024-04-19T02:03:21.677 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2740000000000007e-06 2024-04-19T02:03:21.677 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.677 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.677 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.677 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.137421+0000", 2024-04-19T02:03:21.677 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7782999999999999e-05 2024-04-19T02:03:21.677 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.677 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.677 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.677 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.137481+0000", 2024-04-19T02:03:21.677 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0652000000000001e-05 2024-04-19T02:03:21.677 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.677 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.677 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.677 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.137598+0000", 2024-04-19T02:03:21.678 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011613299999999999 2024-04-19T02:03:21.678 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.678 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.678 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:21.678 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.137654+0000", 2024-04-19T02:03:21.678 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6390000000000001e-05 2024-04-19T02:03:21.678 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.678 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.678 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:21.678 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.143681+0000", 2024-04-19T02:03:21.678 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0060272140000000004 2024-04-19T02:03:21.678 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.678 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.678 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.679 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.143794+0000", 2024-04-19T02:03:21.679 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000112712 2024-04-19T02:03:21.679 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.679 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.679 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.679 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.679 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.679 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:86 1.0 e19/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'56, mlcod=19'51)", 2024-04-19T02:03:21.679 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:00.138559+0000", 2024-04-19T02:03:21.679 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 21.518356300000001, 2024-04-19T02:03:21.679 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0053320930000000004, 2024-04-19T02:03:21.679 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.679 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.679 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:21.679 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.680 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.680 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.680 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.138559+0000", 2024-04-19T02:03:21.680 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.680 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.680 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.680 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.680 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.138559+0000", 2024-04-19T02:03:21.680 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.680 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.680 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.680 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.680 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.138560+0000", 2024-04-19T02:03:21.680 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.44e-06 2024-04-19T02:03:21.680 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.681 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.681 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.681 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.138570+0000", 2024-04-19T02:03:21.681 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6139999999999998e-06 2024-04-19T02:03:21.681 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.681 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.681 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.681 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.138598+0000", 2024-04-19T02:03:21.681 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8146000000000001e-05 2024-04-19T02:03:21.681 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.681 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.681 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.681 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.138611+0000", 2024-04-19T02:03:21.681 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2807e-05 2024-04-19T02:03:21.681 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.681 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.682 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.682 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.138698+0000", 2024-04-19T02:03:21.682 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.6872000000000004e-05 2024-04-19T02:03:21.682 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.682 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.682 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:21.682 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.138727+0000", 2024-04-19T02:03:21.682 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8739999999999999e-05 2024-04-19T02:03:21.682 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.682 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.682 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:21.682 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.143800+0000", 2024-04-19T02:03:21.682 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0050737680000000002 2024-04-19T02:03:21.682 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.683 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.683 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.683 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.143891+0000", 2024-04-19T02:03:21.683 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0705999999999995e-05 2024-04-19T02:03:21.683 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.683 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.683 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.683 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.683 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.683 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:87 1.0 e19/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'57, mlcod=19'51)", 2024-04-19T02:03:21.683 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:00.139853+0000", 2024-04-19T02:03:21.683 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 21.517061843, 2024-04-19T02:03:21.683 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0087466390000000005, 2024-04-19T02:03:21.683 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.683 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.684 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:21.684 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.684 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.684 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.684 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.139853+0000", 2024-04-19T02:03:21.684 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.684 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.684 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.684 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.684 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.139853+0000", 2024-04-19T02:03:21.684 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.684 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.684 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.684 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.684 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.139855+0000", 2024-04-19T02:03:21.684 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.733e-06 2024-04-19T02:03:21.685 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.685 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.685 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.685 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.139912+0000", 2024-04-19T02:03:21.685 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6849999999999999e-05 2024-04-19T02:03:21.685 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.685 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.685 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.685 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.139940+0000", 2024-04-19T02:03:21.685 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7883000000000001e-05 2024-04-19T02:03:21.685 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.685 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.685 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.685 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.139953+0000", 2024-04-19T02:03:21.686 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.314e-05 2024-04-19T02:03:21.686 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.686 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.686 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.686 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.140052+0000", 2024-04-19T02:03:21.686 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9246000000000005e-05 2024-04-19T02:03:21.686 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.686 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.686 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:21.686 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.140084+0000", 2024-04-19T02:03:21.686 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1409999999999999e-05 2024-04-19T02:03:21.686 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.686 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.686 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:21.686 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.148540+0000", 2024-04-19T02:03:21.687 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0084567979999999997 2024-04-19T02:03:21.687 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.687 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.687 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.687 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.148600+0000", 2024-04-19T02:03:21.687 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9579e-05 2024-04-19T02:03:21.687 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.687 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.687 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.687 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.687 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.687 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:88 1.0 e19/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'58, mlcod=19'52)", 2024-04-19T02:03:21.687 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:00.141581+0000", 2024-04-19T02:03:21.687 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 21.515334016000001, 2024-04-19T02:03:21.687 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0070620880000000002, 2024-04-19T02:03:21.688 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.688 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.688 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:21.688 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.688 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.688 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.688 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.141581+0000", 2024-04-19T02:03:21.688 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.688 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.688 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.688 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.688 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.141581+0000", 2024-04-19T02:03:21.688 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.688 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.688 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.689 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.689 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.141582+0000", 2024-04-19T02:03:21.689 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.147e-06 2024-04-19T02:03:21.689 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.689 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.689 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.689 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.141590+0000", 2024-04-19T02:03:21.689 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4229999999999999e-06 2024-04-19T02:03:21.689 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.689 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.689 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.690 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.141609+0000", 2024-04-19T02:03:21.690 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9357000000000001e-05 2024-04-19T02:03:21.690 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.690 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.690 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.690 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.141642+0000", 2024-04-19T02:03:21.691 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.2342999999999998e-05 2024-04-19T02:03:21.691 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.691 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.691 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.691 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.141734+0000", 2024-04-19T02:03:21.691 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.2091999999999999e-05 2024-04-19T02:03:21.692 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.692 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.692 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:21.692 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.141771+0000", 2024-04-19T02:03:21.692 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6913999999999999e-05 2024-04-19T02:03:21.692 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.692 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.693 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:21.693 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.148606+0000", 2024-04-19T02:03:21.693 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0068350030000000001 2024-04-19T02:03:21.693 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.693 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.693 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.693 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.148643+0000", 2024-04-19T02:03:21.694 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7808999999999997e-05 2024-04-19T02:03:21.694 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.694 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.694 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.694 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.694 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.694 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:89 1.0 e19/18 1:30306672:devicehealth::main.db.0000000000000000:head v 19'59, mlcod=19'57)", 2024-04-19T02:03:21.695 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:00.151382+0000", 2024-04-19T02:03:21.695 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 21.505533724999999, 2024-04-19T02:03:21.695 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0037983600000000002, 2024-04-19T02:03:21.695 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.695 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.695 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:21.696 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.696 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.696 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.696 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.151382+0000", 2024-04-19T02:03:21.696 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.696 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.696 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.697 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.697 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.151382+0000", 2024-04-19T02:03:21.697 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.697 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.697 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.697 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.698 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.151383+0000", 2024-04-19T02:03:21.698 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6530000000000001e-06 2024-04-19T02:03:21.698 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.698 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.698 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.698 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.151499+0000", 2024-04-19T02:03:21.699 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000116076 2024-04-19T02:03:21.699 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.699 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.699 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.699 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.151527+0000", 2024-04-19T02:03:21.699 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.794e-05 2024-04-19T02:03:21.700 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.700 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.700 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.700 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.151540+0000", 2024-04-19T02:03:21.700 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3003000000000001e-05 2024-04-19T02:03:21.700 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.701 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.701 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.701 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.151653+0000", 2024-04-19T02:03:21.701 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000112726 2024-04-19T02:03:21.701 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.701 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.702 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:21.702 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.151688+0000", 2024-04-19T02:03:21.702 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5403e-05 2024-04-19T02:03:21.702 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.702 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.702 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:21.703 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.155069+0000", 2024-04-19T02:03:21.703 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0033803570000000001 2024-04-19T02:03:21.703 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.703 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.703 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.703 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.155180+0000", 2024-04-19T02:03:21.703 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000111202 2024-04-19T02:03:21.704 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.704 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.704 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.704 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.704 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.704 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:106 1.0 e19/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'68, mlcod=19'64)", 2024-04-19T02:03:21.704 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:00.314828+0000", 2024-04-19T02:03:21.705 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 21.342086890000001, 2024-04-19T02:03:21.705 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0045193550000000001, 2024-04-19T02:03:21.705 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.705 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.705 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:21.705 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.706 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.706 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.706 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.314828+0000", 2024-04-19T02:03:21.706 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.706 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.706 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.707 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.707 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.314828+0000", 2024-04-19T02:03:21.707 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.707 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.707 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.707 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.707 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.314830+0000", 2024-04-19T02:03:21.708 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.733e-06 2024-04-19T02:03:21.708 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.708 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.708 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.708 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.314889+0000", 2024-04-19T02:03:21.708 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8616000000000003e-05 2024-04-19T02:03:21.708 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.708 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.709 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.709 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.314917+0000", 2024-04-19T02:03:21.709 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8496999999999998e-05 2024-04-19T02:03:21.709 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.709 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.709 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.709 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.314930+0000", 2024-04-19T02:03:21.709 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2607e-05 2024-04-19T02:03:21.710 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.710 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.710 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.710 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.315030+0000", 2024-04-19T02:03:21.710 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010015500000000001 2024-04-19T02:03:21.710 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.710 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.710 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:21.710 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.315062+0000", 2024-04-19T02:03:21.710 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1659999999999998e-05 2024-04-19T02:03:21.710 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.710 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.711 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:21.711 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.319250+0000", 2024-04-19T02:03:21.711 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0041880779999999996 2024-04-19T02:03:21.711 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.711 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.711 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.711 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.319348+0000", 2024-04-19T02:03:21.711 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8009000000000003e-05 2024-04-19T02:03:21.711 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.711 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.711 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.711 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.711 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.712 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:107 1.0 e19/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'69, mlcod=19'64)", 2024-04-19T02:03:21.712 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:00.315743+0000", 2024-04-19T02:03:21.712 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 21.341171877000001, 2024-04-19T02:03:21.712 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003676178, 2024-04-19T02:03:21.712 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.712 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.712 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:21.712 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.712 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.712 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.712 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.315743+0000", 2024-04-19T02:03:21.712 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.713 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.713 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.713 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.713 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.315743+0000", 2024-04-19T02:03:21.713 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.713 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.713 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.713 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.713 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.315745+0000", 2024-04-19T02:03:21.713 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.73e-06 2024-04-19T02:03:21.713 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.713 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.713 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.713 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.315757+0000", 2024-04-19T02:03:21.713 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2089999999999999e-05 2024-04-19T02:03:21.714 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.714 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.714 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.714 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.315786+0000", 2024-04-19T02:03:21.714 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8333000000000001e-05 2024-04-19T02:03:21.714 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.714 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.714 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.714 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.315799+0000", 2024-04-19T02:03:21.714 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3013000000000001e-05 2024-04-19T02:03:21.714 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.714 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.714 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.715 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.315889+0000", 2024-04-19T02:03:21.715 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0296000000000004e-05 2024-04-19T02:03:21.715 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.715 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.715 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:21.715 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.315916+0000", 2024-04-19T02:03:21.715 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7596000000000001e-05 2024-04-19T02:03:21.715 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.715 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.715 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:21.715 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.319353+0000", 2024-04-19T02:03:21.715 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0034365440000000001 2024-04-19T02:03:21.716 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.716 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.716 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.716 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.319420+0000", 2024-04-19T02:03:21.716 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6575999999999994e-05 2024-04-19T02:03:21.716 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.716 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.716 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.716 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.716 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.716 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:113 1.0 e19/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'75, mlcod=19'71)", 2024-04-19T02:03:21.716 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:00.323587+0000", 2024-04-19T02:03:21.716 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 21.333328558000002, 2024-04-19T02:03:21.717 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0058327889999999997, 2024-04-19T02:03:21.717 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.717 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.717 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:21.717 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.717 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.717 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.717 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.323587+0000", 2024-04-19T02:03:21.717 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.717 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.718 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.718 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.718 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.323587+0000", 2024-04-19T02:03:21.718 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.718 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.718 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.718 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.718 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.323589+0000", 2024-04-19T02:03:21.718 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.934e-06 2024-04-19T02:03:21.718 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.718 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.718 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.718 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.323601+0000", 2024-04-19T02:03:21.718 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2683e-05 2024-04-19T02:03:21.719 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.719 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.719 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.719 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.323629+0000", 2024-04-19T02:03:21.719 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7823000000000001e-05 2024-04-19T02:03:21.719 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.719 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.719 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.719 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.323642+0000", 2024-04-19T02:03:21.720 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3169999999999999e-05 2024-04-19T02:03:21.720 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.720 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.720 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.720 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.323762+0000", 2024-04-19T02:03:21.720 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011994600000000001 2024-04-19T02:03:21.720 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.721 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.721 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:21.721 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.323814+0000", 2024-04-19T02:03:21.721 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1430000000000001e-05 2024-04-19T02:03:21.721 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.721 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.721 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:21.722 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.329316+0000", 2024-04-19T02:03:21.722 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00550257 2024-04-19T02:03:21.722 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.722 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.722 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.722 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.329419+0000", 2024-04-19T02:03:21.722 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010323300000000001 2024-04-19T02:03:21.722 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.722 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.722 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.722 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.722 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.722 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:114 1.0 e19/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'76, mlcod=19'72)", 2024-04-19T02:03:21.722 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:00.324817+0000", 2024-04-19T02:03:21.722 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 21.332098512999998, 2024-04-19T02:03:21.723 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0057275679999999997, 2024-04-19T02:03:21.723 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.723 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.723 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:21.723 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.723 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.723 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.723 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.324817+0000", 2024-04-19T02:03:21.723 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.723 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.723 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.724 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.724 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.324817+0000", 2024-04-19T02:03:21.724 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.724 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.724 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.724 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.724 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.324818+0000", 2024-04-19T02:03:21.724 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6559999999999999e-06 2024-04-19T02:03:21.724 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.724 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.724 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.724 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.324831+0000", 2024-04-19T02:03:21.724 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.242e-05 2024-04-19T02:03:21.724 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.724 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.725 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.725 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.324855+0000", 2024-04-19T02:03:21.725 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4300000000000001e-05 2024-04-19T02:03:21.725 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.725 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.725 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.725 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.324869+0000", 2024-04-19T02:03:21.725 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3713000000000001e-05 2024-04-19T02:03:21.725 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.725 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.725 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.725 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.324973+0000", 2024-04-19T02:03:21.725 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000104183 2024-04-19T02:03:21.725 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.725 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.726 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:21.726 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.325009+0000", 2024-04-19T02:03:21.726 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5698999999999998e-05 2024-04-19T02:03:21.726 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.726 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.726 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:21.726 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.330447+0000", 2024-04-19T02:03:21.726 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0054381949999999998 2024-04-19T02:03:21.726 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.726 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.726 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.726 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.330544+0000", 2024-04-19T02:03:21.726 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7402e-05 2024-04-19T02:03:21.726 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.726 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.727 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.727 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.727 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.727 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:115 1.0 e19/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'77, mlcod=19'72)", 2024-04-19T02:03:21.727 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:00.325999+0000", 2024-04-19T02:03:21.727 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 21.330916042999998, 2024-04-19T02:03:21.727 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0046238750000000004, 2024-04-19T02:03:21.727 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.727 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.727 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:21.727 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.727 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.727 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.727 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.325999+0000", 2024-04-19T02:03:21.727 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.728 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.728 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.728 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.728 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.325999+0000", 2024-04-19T02:03:21.728 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.728 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.728 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.728 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.728 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.326001+0000", 2024-04-19T02:03:21.728 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.73e-06 2024-04-19T02:03:21.728 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.728 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.728 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.728 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.326057+0000", 2024-04-19T02:03:21.728 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5723000000000002e-05 2024-04-19T02:03:21.729 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.729 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.729 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.729 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.326085+0000", 2024-04-19T02:03:21.729 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8197000000000001e-05 2024-04-19T02:03:21.729 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.729 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.729 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.729 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.326098+0000", 2024-04-19T02:03:21.729 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2819999999999999e-05 2024-04-19T02:03:21.729 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.729 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.729 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.729 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.328527+0000", 2024-04-19T02:03:21.730 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0024289139999999999 2024-04-19T02:03:21.730 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.730 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.730 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:21.730 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.328557+0000", 2024-04-19T02:03:21.730 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0586000000000002e-05 2024-04-19T02:03:21.730 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.730 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.730 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:21.730 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.330552+0000", 2024-04-19T02:03:21.730 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0019943679999999998 2024-04-19T02:03:21.730 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.730 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.730 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.731 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.330623+0000", 2024-04-19T02:03:21.731 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1537000000000003e-05 2024-04-19T02:03:21.731 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.731 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.731 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.731 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.731 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.731 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:117 1.0 e19/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'79, mlcod=19'73)", 2024-04-19T02:03:21.731 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:00.328486+0000", 2024-04-19T02:03:21.731 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 21.328429222, 2024-04-19T02:03:21.731 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0051412910000000001, 2024-04-19T02:03:21.731 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.731 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.731 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:21.731 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.732 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.732 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.732 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.328486+0000", 2024-04-19T02:03:21.732 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.732 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.732 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.732 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.732 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.328486+0000", 2024-04-19T02:03:21.732 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.732 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.732 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.732 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.732 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.328487+0000", 2024-04-19T02:03:21.732 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0359999999999999e-06 2024-04-19T02:03:21.732 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.732 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.733 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.733 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.328494+0000", 2024-04-19T02:03:21.733 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4139999999999997e-06 2024-04-19T02:03:21.733 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.733 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.733 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.733 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.328508+0000", 2024-04-19T02:03:21.733 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.308e-05 2024-04-19T02:03:21.733 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.733 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.733 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.734 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.328516+0000", 2024-04-19T02:03:21.734 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7960000000000001e-06 2024-04-19T02:03:21.734 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.734 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.734 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.734 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.329976+0000", 2024-04-19T02:03:21.734 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0014596089999999999 2024-04-19T02:03:21.734 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.734 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.734 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:21.734 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.330029+0000", 2024-04-19T02:03:21.734 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2899000000000003e-05 2024-04-19T02:03:21.734 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.734 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.735 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:21.735 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.333536+0000", 2024-04-19T02:03:21.735 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0035075380000000001 2024-04-19T02:03:21.735 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.735 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.735 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.735 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.333627+0000", 2024-04-19T02:03:21.735 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0919000000000001e-05 2024-04-19T02:03:21.735 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.735 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.735 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.735 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.735 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.735 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:119 1.0 e19/18 1:30306672:devicehealth::main.db.0000000000000000:head v 19'81, mlcod=19'78)", 2024-04-19T02:03:21.735 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:00.336987+0000", 2024-04-19T02:03:21.735 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 21.319927941, 2024-04-19T02:03:21.736 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0036515050000000002, 2024-04-19T02:03:21.736 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.736 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.736 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:21.736 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.736 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.736 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.736 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.336987+0000", 2024-04-19T02:03:21.736 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.736 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.736 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.736 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.736 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.336987+0000", 2024-04-19T02:03:21.736 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.737 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.737 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.737 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.737 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.336989+0000", 2024-04-19T02:03:21.737 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7540000000000001e-06 2024-04-19T02:03:21.737 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.737 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.737 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.737 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.337107+0000", 2024-04-19T02:03:21.737 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000117559 2024-04-19T02:03:21.737 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.737 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.738 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.738 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.337131+0000", 2024-04-19T02:03:21.738 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4043e-05 2024-04-19T02:03:21.738 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.738 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.738 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.738 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.337143+0000", 2024-04-19T02:03:21.738 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2737e-05 2024-04-19T02:03:21.738 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.738 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.738 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.738 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.337266+0000", 2024-04-19T02:03:21.738 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012231500000000001 2024-04-19T02:03:21.738 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.739 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.739 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:21.739 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.337325+0000", 2024-04-19T02:03:21.739 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9406e-05 2024-04-19T02:03:21.739 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.739 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.739 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:21.739 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.340539+0000", 2024-04-19T02:03:21.739 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003214242 2024-04-19T02:03:21.739 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.739 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.739 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.739 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:00.340639+0000", 2024-04-19T02:03:21.739 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9449000000000003e-05 2024-04-19T02:03:21.740 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.740 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.740 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.740 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.740 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.740 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:160 1.0 e19/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'114, mlcod=19'108)", 2024-04-19T02:03:21.740 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:01.297268+0000", 2024-04-19T02:03:21.740 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 20.359647396, 2024-04-19T02:03:21.740 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0044380230000000001, 2024-04-19T02:03:21.740 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.740 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.740 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:21.740 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.740 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.741 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.741 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.297268+0000", 2024-04-19T02:03:21.741 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.741 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.741 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.741 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.741 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.297268+0000", 2024-04-19T02:03:21.741 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.741 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.741 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.741 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.741 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.297269+0000", 2024-04-19T02:03:21.741 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.61e-06 2024-04-19T02:03:21.741 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.741 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.742 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.742 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.297282+0000", 2024-04-19T02:03:21.742 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.279e-05 2024-04-19T02:03:21.742 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.742 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.742 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.742 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.297312+0000", 2024-04-19T02:03:21.742 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0015999999999999e-05 2024-04-19T02:03:21.742 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.742 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.742 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.742 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.297329+0000", 2024-04-19T02:03:21.742 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6843000000000001e-05 2024-04-19T02:03:21.742 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.743 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.743 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.743 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.297412+0000", 2024-04-19T02:03:21.743 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2646999999999997e-05 2024-04-19T02:03:21.743 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.743 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.743 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:21.743 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.297439+0000", 2024-04-19T02:03:21.743 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7259000000000001e-05 2024-04-19T02:03:21.743 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.743 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.743 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:21.743 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.301612+0000", 2024-04-19T02:03:21.744 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0041734190000000003 2024-04-19T02:03:21.744 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.744 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.744 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.744 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.301706+0000", 2024-04-19T02:03:21.744 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3438999999999998e-05 2024-04-19T02:03:21.744 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.744 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.744 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.744 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.744 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.744 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:161 1.0 e19/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'115, mlcod=19'111)", 2024-04-19T02:03:21.744 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:01.298463+0000", 2024-04-19T02:03:21.745 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 20.358452692, 2024-04-19T02:03:21.745 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0051605569999999996, 2024-04-19T02:03:21.745 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.745 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.745 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:21.745 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.745 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.745 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.745 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.298463+0000", 2024-04-19T02:03:21.745 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.745 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.745 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.745 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.745 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.298463+0000", 2024-04-19T02:03:21.745 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.745 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.746 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.746 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.746 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.298464+0000", 2024-04-19T02:03:21.746 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.72e-06 2024-04-19T02:03:21.746 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.746 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.746 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.746 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.298577+0000", 2024-04-19T02:03:21.746 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000113083 2024-04-19T02:03:21.746 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.746 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.746 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.746 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.298608+0000", 2024-04-19T02:03:21.746 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1115999999999998e-05 2024-04-19T02:03:21.747 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.747 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.747 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.747 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.298633+0000", 2024-04-19T02:03:21.747 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4542999999999999e-05 2024-04-19T02:03:21.747 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.747 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.747 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.747 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.298761+0000", 2024-04-19T02:03:21.747 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000128026 2024-04-19T02:03:21.747 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.747 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.747 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:21.747 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.298817+0000", 2024-04-19T02:03:21.747 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5495999999999999e-05 2024-04-19T02:03:21.748 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.748 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.748 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:21.748 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.303529+0000", 2024-04-19T02:03:21.748 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0047123970000000001 2024-04-19T02:03:21.748 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.748 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.748 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.748 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.303623+0000", 2024-04-19T02:03:21.748 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4176000000000001e-05 2024-04-19T02:03:21.748 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.748 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.748 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.748 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.748 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.748 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:162 1.0 e19/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'116, mlcod=19'111)", 2024-04-19T02:03:21.749 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:01.299115+0000", 2024-04-19T02:03:21.749 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 20.35779986, 2024-04-19T02:03:21.749 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0045818350000000002, 2024-04-19T02:03:21.749 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.749 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.749 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:21.749 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.749 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.749 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.749 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.299115+0000", 2024-04-19T02:03:21.749 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.749 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.749 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.749 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.750 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.299115+0000", 2024-04-19T02:03:21.750 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.750 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.750 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.750 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.750 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.299117+0000", 2024-04-19T02:03:21.750 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4270000000000001e-06 2024-04-19T02:03:21.750 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.750 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.750 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.750 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.299126+0000", 2024-04-19T02:03:21.750 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6630000000000005e-06 2024-04-19T02:03:21.750 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.750 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.750 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.751 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.299143+0000", 2024-04-19T02:03:21.751 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6793e-05 2024-04-19T02:03:21.751 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.751 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.751 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.751 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.299154+0000", 2024-04-19T02:03:21.751 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0273e-05 2024-04-19T02:03:21.751 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.751 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.751 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.751 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.300143+0000", 2024-04-19T02:03:21.751 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00098952299999999997 2024-04-19T02:03:21.751 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.751 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.751 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:21.752 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.300175+0000", 2024-04-19T02:03:21.752 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.201e-05 2024-04-19T02:03:21.752 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.752 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.752 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:21.752 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.303627+0000", 2024-04-19T02:03:21.752 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003451866 2024-04-19T02:03:21.752 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.752 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.752 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.752 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.303697+0000", 2024-04-19T02:03:21.752 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0279999999999998e-05 2024-04-19T02:03:21.752 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.752 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.752 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.752 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.753 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.753 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:163 1.0 e19/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'117, mlcod=19'112)", 2024-04-19T02:03:21.753 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:01.299976+0000", 2024-04-19T02:03:21.753 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 20.356939050000001, 2024-04-19T02:03:21.753 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0037995070000000001, 2024-04-19T02:03:21.753 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.753 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.753 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:21.753 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.753 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.753 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.753 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.299976+0000", 2024-04-19T02:03:21.753 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.753 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.753 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.754 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.754 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.299976+0000", 2024-04-19T02:03:21.754 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.754 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.754 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.754 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.754 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.299977+0000", 2024-04-19T02:03:21.754 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.17e-06 2024-04-19T02:03:21.754 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.754 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.754 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.755 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.299991+0000", 2024-04-19T02:03:21.755 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.324e-05 2024-04-19T02:03:21.755 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.755 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.755 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.755 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.300009+0000", 2024-04-19T02:03:21.755 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8490000000000001e-05 2024-04-19T02:03:21.755 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.755 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.755 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.755 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.300019+0000", 2024-04-19T02:03:21.755 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6430000000000007e-06 2024-04-19T02:03:21.756 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.756 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.756 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.756 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.300687+0000", 2024-04-19T02:03:21.756 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00066863800000000004 2024-04-19T02:03:21.756 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.756 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.756 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:21.756 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.300748+0000", 2024-04-19T02:03:21.756 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0112999999999999e-05 2024-04-19T02:03:21.756 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.756 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.756 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:21.756 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.303704+0000", 2024-04-19T02:03:21.757 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0029559970000000001 2024-04-19T02:03:21.757 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.757 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.757 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.757 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.303776+0000", 2024-04-19T02:03:21.757 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2216000000000001e-05 2024-04-19T02:03:21.757 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.757 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.757 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.757 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.757 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.757 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_repop(client.4207.0:179 1.0 e21/18 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'129, mlcod=19'125)", 2024-04-19T02:03:21.757 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:01.981445+0000", 2024-04-19T02:03:21.757 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 19.675470431000001, 2024-04-19T02:03:21.757 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0035471220000000002, 2024-04-19T02:03:21.758 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:21.758 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:21.758 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:21.758 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:21.758 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.758 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:21.758 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.981445+0000", 2024-04-19T02:03:21.758 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.758 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.758 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.758 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:21.758 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.981445+0000", 2024-04-19T02:03:21.759 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:21.759 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.759 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.759 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:21.759 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.981446+0000", 2024-04-19T02:03:21.759 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9699999999999994e-07 2024-04-19T02:03:21.759 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.759 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.759 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:21.759 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.981641+0000", 2024-04-19T02:03:21.759 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00019564499999999999 2024-04-19T02:03:21.759 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.759 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.759 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:21.759 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.981658+0000", 2024-04-19T02:03:21.760 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6296000000000001e-05 2024-04-19T02:03:21.760 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.760 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.760 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:21.760 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.981668+0000", 2024-04-19T02:03:21.760 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0346999999999999e-05 2024-04-19T02:03:21.760 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.760 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.760 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:21.760 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.982459+0000", 2024-04-19T02:03:21.760 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00079046400000000005 2024-04-19T02:03:21.760 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.760 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.760 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:21.760 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.982511+0000", 2024-04-19T02:03:21.760 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2562999999999998e-05 2024-04-19T02:03:21.761 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.761 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.761 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:21.761 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.984939+0000", 2024-04-19T02:03:21.761 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0024275049999999999 2024-04-19T02:03:21.761 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:21.761 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:21.761 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:21.761 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:01.984992+0000", 2024-04-19T02:03:21.761 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3304999999999999e-05 2024-04-19T02:03:21.761 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.761 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.761 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.761 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:21.761 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:21.761 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:21.764 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-19T02:03:21.765 INFO:tasks.rados.rados.0.smithi059.stdout:114: done (0 left) 2024-04-19T02:03:21.765 INFO:tasks.rados.rados.0.smithi059.stdout:115: delete oid 15 current snap is 5 2024-04-19T02:03:21.765 INFO:tasks.rados.rados.0.smithi059.stdout:115: done (0 left) 2024-04-19T02:03:21.765 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 18 current snap is 5 2024-04-19T02:03:21.765 INFO:tasks.rados.rados.0.smithi059.stdout:116: seq_num 71 ranges {294912=425984,720896=573440,1294336=196608} 2024-04-19T02:03:21.765 INFO:tasks.rados.rados.0.smithi059.stdout:116: writing smithi05930959-18 from 294912 to 720896 tid 1 2024-04-19T02:03:21.765 INFO:tasks.rados.rados.0.smithi059.stdout:116: writing smithi05930959-18 from 720896 to 1294336 tid 2 2024-04-19T02:03:21.765 INFO:tasks.rados.rados.0.smithi059.stdout:116: writing smithi05930959-18 from 1294336 to 1490944 tid 3 2024-04-19T02:03:21.765 INFO:tasks.rados.rados.0.smithi059.stdout:117: snap_create 2024-04-19T02:03:21.765 INFO:tasks.rados.rados.0.smithi059.stdout:116: finishing write tid 1 to smithi05930959-18 2024-04-19T02:03:21.765 INFO:tasks.rados.rados.0.smithi059.stdout:116: finishing write tid 2 to smithi05930959-18 2024-04-19T02:03:21.766 INFO:tasks.rados.rados.0.smithi059.stdout:116: finishing write tid 3 to smithi05930959-18 2024-04-19T02:03:21.766 INFO:tasks.rados.rados.0.smithi059.stdout:116: finishing write tid 4 to smithi05930959-18 2024-04-19T02:03:21.766 INFO:tasks.rados.rados.0.smithi059.stdout:116: finishing write tid 5 to smithi05930959-18 2024-04-19T02:03:21.766 INFO:tasks.rados.rados.0.smithi059.stdout:116: oid 18 updating version 0 to 31 2024-04-19T02:03:21.766 INFO:tasks.rados.rados.0.smithi059.stdout:116: oid 18 version 31 is already newer than 30 2024-04-19T02:03:21.766 INFO:tasks.rados.rados.0.smithi059.stdout:116: oid 18 version 31 is already newer than 28 2024-04-19T02:03:21.766 INFO:tasks.rados.rados.0.smithi059.stdout:116: oid 18 version 31 is already newer than 29 2024-04-19T02:03:21.766 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 18 v 31 (ObjNum 71 snap 5 seq_num 71) dirty exists 2024-04-19T02:03:21.766 INFO:tasks.rados.rados.0.smithi059.stdout:116: left oid 18 (ObjNum 71 snap 5 seq_num 71) 2024-04-19T02:03:21.767 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:21.763+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:21.864 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:21.863+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:21.875 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T02:03:21.893 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:21.893 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:21.893 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 0 2024-04-19T02:03:21.893 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:21.907 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-19T02:03:21.965 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:21.963+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:22.035 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-04-19T02:03:22.035 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T02:03:22.052 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-04-19T02:03:22.052 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T02:03:22.067 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:22.067+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:22.067 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:22.068 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:22.068 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-04-19T02:03:22.068 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-04-19T02:03:22.068 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:22.071 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-04-19T02:03:22.071 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T02:03:22.082 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-19T02:03:22.089 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-04-19T02:03:22.089 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T02:03:22.109 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-04-19T02:03:22.109 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T02:03:22.128 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-04-19T02:03:22.128 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T02:03:22.147 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-04-19T02:03:22.148 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T02:03:22.167 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:22.167+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:22.168 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-04-19T02:03:22.168 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T02:03:22.190 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-04-19T02:03:22.190 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T02:03:22.200 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:22.200 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:22.211 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-04-19T02:03:22.212 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T02:03:22.214 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-19T02:03:22.234 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-04-19T02:03:22.234 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T02:03:22.257 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-04-19T02:03:22.257 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T02:03:22.268 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:22.267+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:22.337 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:22.337 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:22.351 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-19T02:03:22.369 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:22.367+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:22.470 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:22.471+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:22.511 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:22.511 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:22.525 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-19T02:03:22.570 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:22.571+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:22.641 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:22.641 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:22.655 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-19T02:03:22.698 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:22.699+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:22.700 INFO:tasks.rados.rados.0.smithi059.stdout:116: done (1 left) 2024-04-19T02:03:22.701 INFO:tasks.rados.rados.0.smithi059.stdout:117: done (0 left) 2024-04-19T02:03:22.701 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 29 current snap is 6 2024-04-19T02:03:22.701 INFO:tasks.rados.rados.0.smithi059.stdout:118: seq_num 72 ranges {475136=442368,917504=720896,1638400=49152} 2024-04-19T02:03:22.706 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-19T02:03:22.716 INFO:tasks.rados.rados.0.smithi059.stdout:118: writing smithi05930959-29 from 475136 to 917504 tid 1 2024-04-19T02:03:22.731 INFO:tasks.rados.rados.0.smithi059.stdout:118: writing smithi05930959-29 from 917504 to 1638400 tid 2 2024-04-19T02:03:22.732 INFO:tasks.rados.rados.0.smithi059.stdout:118: writing smithi05930959-29 from 1638400 to 1687552 tid 3 2024-04-19T02:03:22.732 INFO:tasks.rados.rados.0.smithi059.stdout:119: rollback oid 43 current snap is 6 2024-04-19T02:03:22.732 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 43 to 4 2024-04-19T02:03:22.732 INFO:tasks.rados.rados.0.smithi059.stdout:118: finishing write tid 1 to smithi05930959-29 2024-04-19T02:03:22.732 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 7 current snap is 6 2024-04-19T02:03:22.732 INFO:tasks.rados.rados.0.smithi059.stdout:120: seq_num 73 ranges {114688=458752,573440=475136,1048576=294912} 2024-04-19T02:03:22.746 INFO:tasks.rados.rados.0.smithi059.stdout:120: writing smithi05930959-7 from 114688 to 573440 tid 1 2024-04-19T02:03:22.754 INFO:tasks.rados.rados.0.smithi059.stdout:120: writing smithi05930959-7 from 573440 to 1048576 tid 2 2024-04-19T02:03:22.758 INFO:tasks.rados.rados.0.smithi059.stdout:120: writing smithi05930959-7 from 1048576 to 1343488 tid 3 2024-04-19T02:03:22.759 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 17 current snap is 6 2024-04-19T02:03:22.759 INFO:tasks.rados.rados.0.smithi059.stdout:121: seq_num 74 ranges {0=475136,475136=622592,1097728=147456} 2024-04-19T02:03:22.765 INFO:tasks.rados.rados.0.smithi059.stdout:121: writing smithi05930959-17 from 0 to 475136 tid 1 2024-04-19T02:03:22.772 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:22.771+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:22.772 INFO:tasks.rados.rados.0.smithi059.stdout:121: writing smithi05930959-17 from 475136 to 1097728 tid 2 2024-04-19T02:03:22.775 INFO:tasks.rados.rados.0.smithi059.stdout:121: writing smithi05930959-17 from 1097728 to 1245184 tid 3 2024-04-19T02:03:22.775 INFO:tasks.rados.rados.0.smithi059.stdout:122: read oid 21 snap 3 2024-04-19T02:03:22.775 INFO:tasks.rados.rados.0.smithi059.stdout:122: expect (ObjNum 20 snap 0 seq_num 20) 2024-04-19T02:03:22.775 INFO:tasks.rados.rados.0.smithi059.stdout:123: delete oid 47 current snap is 6 2024-04-19T02:03:22.775 INFO:tasks.rados.rados.0.smithi059.stdout:118: finishing write tid 2 to smithi05930959-29 2024-04-19T02:03:22.775 INFO:tasks.rados.rados.0.smithi059.stdout:118: finishing write tid 3 to smithi05930959-29 2024-04-19T02:03:22.776 INFO:tasks.rados.rados.0.smithi059.stdout:118: finishing write tid 4 to smithi05930959-29 2024-04-19T02:03:22.776 INFO:tasks.rados.rados.0.smithi059.stdout:119: finishing rollback tid 0 to smithi05930959-43 2024-04-19T02:03:22.776 INFO:tasks.rados.rados.0.smithi059.stdout:119: finishing rollback tid 1 to smithi05930959-43 2024-04-19T02:03:22.776 INFO:tasks.rados.rados.0.smithi059.stdout:119: finishing rollback tid 2 to smithi05930959-43 2024-04-19T02:03:22.776 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 43 v 13 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-19T02:03:22.776 INFO:tasks.rados.rados.0.smithi059.stdout:118: finishing write tid 5 to smithi05930959-29 2024-04-19T02:03:22.776 INFO:tasks.rados.rados.0.smithi059.stdout:118: oid 29 updating version 0 to 22 2024-04-19T02:03:22.777 INFO:tasks.rados.rados.0.smithi059.stdout:118: oid 29 version 22 is already newer than 21 2024-04-19T02:03:22.777 INFO:tasks.rados.rados.0.smithi059.stdout:118: oid 29 version 22 is already newer than 20 2024-04-19T02:03:22.777 INFO:tasks.rados.rados.0.smithi059.stdout:118: oid 29 version 22 is already newer than 19 2024-04-19T02:03:22.777 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 29 v 22 (ObjNum 72 snap 6 seq_num 72) dirty exists 2024-04-19T02:03:22.777 INFO:tasks.rados.rados.0.smithi059.stdout:118: left oid 29 (ObjNum 72 snap 6 seq_num 72) 2024-04-19T02:03:22.777 INFO:tasks.rados.rados.0.smithi059.stdout:120: finishing write tid 1 to smithi05930959-7 2024-04-19T02:03:22.777 INFO:tasks.rados.rados.0.smithi059.stdout:120: finishing write tid 2 to smithi05930959-7 2024-04-19T02:03:22.777 INFO:tasks.rados.rados.0.smithi059.stdout:120: finishing write tid 3 to smithi05930959-7 2024-04-19T02:03:22.777 INFO:tasks.rados.rados.0.smithi059.stdout:120: finishing write tid 4 to smithi05930959-7 2024-04-19T02:03:22.777 INFO:tasks.rados.rados.0.smithi059.stdout:120: finishing write tid 5 to smithi05930959-7 2024-04-19T02:03:22.777 INFO:tasks.rados.rados.0.smithi059.stdout:120: oid 7 updating version 0 to 21 2024-04-19T02:03:22.777 INFO:tasks.rados.rados.0.smithi059.stdout:120: oid 7 updating version 21 to 22 2024-04-19T02:03:22.777 INFO:tasks.rados.rados.0.smithi059.stdout:120: oid 7 version 22 is already newer than 20 2024-04-19T02:03:22.778 INFO:tasks.rados.rados.0.smithi059.stdout:120: oid 7 version 22 is already newer than 19 2024-04-19T02:03:22.778 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 7 v 22 (ObjNum 73 snap 6 seq_num 73) dirty exists 2024-04-19T02:03:22.778 INFO:tasks.rados.rados.0.smithi059.stdout:120: left oid 7 (ObjNum 73 snap 6 seq_num 73) 2024-04-19T02:03:22.778 INFO:tasks.rados.rados.0.smithi059.stdout:121: finishing write tid 1 to smithi05930959-17 2024-04-19T02:03:22.783 INFO:tasks.rados.rados.0.smithi059.stdout:121: finishing write tid 2 to smithi05930959-17 2024-04-19T02:03:22.785 INFO:tasks.rados.rados.0.smithi059.stdout:118: done (5 left) 2024-04-19T02:03:22.785 INFO:tasks.rados.rados.0.smithi059.stdout:119: done (4 left) 2024-04-19T02:03:22.785 INFO:tasks.rados.rados.0.smithi059.stdout:120: done (3 left) 2024-04-19T02:03:22.785 INFO:tasks.rados.rados.0.smithi059.stdout:123: done (2 left) 2024-04-19T02:03:22.785 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 20 current snap is 6 2024-04-19T02:03:22.785 INFO:tasks.rados.rados.0.smithi059.stdout:124: seq_num 76 ranges {327680=507904,835584=507904,1343488=262144} 2024-04-19T02:03:22.793 INFO:tasks.rados.rados.0.smithi059.stdout:124: writing smithi05930959-20 from 327680 to 835584 tid 1 2024-04-19T02:03:22.798 INFO:tasks.rados.rados.0.smithi059.stdout:124: writing smithi05930959-20 from 835584 to 1343488 tid 2 2024-04-19T02:03:22.800 INFO:tasks.rados.rados.0.smithi059.stdout:124: writing smithi05930959-20 from 1343488 to 1605632 tid 3 2024-04-19T02:03:22.801 INFO:tasks.rados.rados.0.smithi059.stdout:125: delete oid 38 current snap is 6 2024-04-19T02:03:22.806 INFO:tasks.rados.rados.0.smithi059.stdout:121: finishing write tid 3 to smithi05930959-17 2024-04-19T02:03:22.806 INFO:tasks.rados.rados.0.smithi059.stdout:121: finishing write tid 4 to smithi05930959-17 2024-04-19T02:03:22.806 INFO:tasks.rados.rados.0.smithi059.stdout:122: expect (ObjNum 20 snap 0 seq_num 20) 2024-04-19T02:03:22.810 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:22.810 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:22.817 INFO:tasks.rados.rados.0.smithi059.stdout:121: finishing write tid 5 to smithi05930959-17 2024-04-19T02:03:22.817 INFO:tasks.rados.rados.0.smithi059.stdout:121: oid 17 updating version 0 to 51 2024-04-19T02:03:22.817 INFO:tasks.rados.rados.0.smithi059.stdout:121: oid 17 version 51 is already newer than 49 2024-04-19T02:03:22.817 INFO:tasks.rados.rados.0.smithi059.stdout:121: oid 17 version 51 is already newer than 48 2024-04-19T02:03:22.817 INFO:tasks.rados.rados.0.smithi059.stdout:121: oid 17 version 51 is already newer than 50 2024-04-19T02:03:22.817 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 17 v 51 (ObjNum 74 snap 6 seq_num 74) dirty exists 2024-04-19T02:03:22.817 INFO:tasks.rados.rados.0.smithi059.stdout:121: left oid 17 (ObjNum 74 snap 6 seq_num 74) 2024-04-19T02:03:22.818 INFO:tasks.rados.rados.0.smithi059.stdout:121: done (3 left) 2024-04-19T02:03:22.818 INFO:tasks.rados.rados.0.smithi059.stdout:122: done (2 left) 2024-04-19T02:03:22.818 INFO:tasks.rados.rados.0.smithi059.stdout:125: done (1 left) 2024-04-19T02:03:22.818 INFO:tasks.rados.rados.0.smithi059.stdout:126: snap_remove snap 5 2024-04-19T02:03:22.825 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-19T02:03:22.873 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:22.871+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:22.945 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:22.945 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:22.959 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-19T02:03:22.974 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:22.975+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:23.075 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:23.075+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:23.076 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:23.076 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:23.090 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-19T02:03:23.176 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:23.179+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:23.256 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:23.257 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:23.270 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-19T02:03:23.277 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:03:23.275+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:03:23.378 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:23.379+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:23.436 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:23.436 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:23.450 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-19T02:03:23.493 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:23.495+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:23.580 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:23.579+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:23.611 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:23.611 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:23.625 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-19T02:03:23.681 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:23.683+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:23.698 INFO:teuthology.orchestra.run.smithi059.stderr:nodeep-scrub is set 2024-04-19T02:03:23.704 INFO:tasks.rados.rados.0.smithi059.stdout:126: done (1 left) 2024-04-19T02:03:23.705 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 39 current snap is 6 2024-04-19T02:03:23.705 INFO:tasks.rados.rados.0.smithi059.stdout:127: seq_num 78 ranges {638976=557056,1196032=671744,1867776=98304} 2024-04-19T02:03:23.716 INFO:tasks.rados.rados.0.smithi059.stdout:127: writing smithi05930959-39 from 638976 to 1196032 tid 1 2024-04-19T02:03:23.723 INFO:tasks.rados.rados.0.smithi059.stdout:127: writing smithi05930959-39 from 1196032 to 1867776 tid 2 2024-04-19T02:03:23.725 INFO:tasks.rados.rados.0.smithi059.stdout:127: writing smithi05930959-39 from 1867776 to 1966080 tid 3 2024-04-19T02:03:23.725 INFO:tasks.rados.rados.0.smithi059.stdout:124: finishing write tid 1 to smithi05930959-20 2024-04-19T02:03:23.725 INFO:tasks.rados.rados.0.smithi059.stdout:124: finishing write tid 2 to smithi05930959-20 2024-04-19T02:03:23.725 INFO:tasks.rados.rados.0.smithi059.stdout:124: finishing write tid 3 to smithi05930959-20 2024-04-19T02:03:23.725 INFO:tasks.rados.rados.0.smithi059.stdout:124: finishing write tid 4 to smithi05930959-20 2024-04-19T02:03:23.725 INFO:tasks.rados.rados.0.smithi059.stdout:124: finishing write tid 5 to smithi05930959-20 2024-04-19T02:03:23.725 INFO:tasks.rados.rados.0.smithi059.stdout:124: oid 20 updating version 0 to 28 2024-04-19T02:03:23.725 INFO:tasks.rados.rados.0.smithi059.stdout:124: oid 20 version 28 is already newer than 27 2024-04-19T02:03:23.725 INFO:tasks.rados.rados.0.smithi059.stdout:124: oid 20 updating version 28 to 30 2024-04-19T02:03:23.725 INFO:tasks.rados.rados.0.smithi059.stdout:124: oid 20 version 30 is already newer than 29 2024-04-19T02:03:23.726 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 20 v 30 (ObjNum 76 snap 6 seq_num 76) dirty exists 2024-04-19T02:03:23.726 INFO:tasks.rados.rados.0.smithi059.stdout:124: left oid 20 (ObjNum 76 snap 6 seq_num 76) 2024-04-19T02:03:23.726 INFO:tasks.rados.rados.0.smithi059.stdout:124: done (1 left) 2024-04-19T02:03:23.726 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 27 current snap is 6 2024-04-19T02:03:23.726 INFO:tasks.rados.rados.0.smithi059.stdout:128: seq_num 79 ranges {0=573440,573440=425984,999424=344064} 2024-04-19T02:03:23.731 INFO:tasks.rados.rados.0.smithi059.stdout:128: writing smithi05930959-27 from 0 to 573440 tid 1 2024-04-19T02:03:23.735 INFO:tasks.rados.rados.0.smithi059.stdout:128: writing smithi05930959-27 from 573440 to 999424 tid 2 2024-04-19T02:03:23.739 INFO:tasks.rados.rados.0.smithi059.stdout:128: writing smithi05930959-27 from 999424 to 1343488 tid 3 2024-04-19T02:03:23.739 INFO:tasks.rados.rados.0.smithi059.stdout:129: setattr oid 29 current snap is 6 2024-04-19T02:03:23.740 INFO:tasks.rados.rados.0.smithi059.stdout:127: finishing write tid 1 to smithi05930959-39 2024-04-19T02:03:23.740 INFO:tasks.rados.rados.0.smithi059.stdout:127: finishing write tid 2 to smithi05930959-39 2024-04-19T02:03:23.741 INFO:tasks.rados.rados.0.smithi059.stdout:127: finishing write tid 3 to smithi05930959-39 2024-04-19T02:03:23.742 INFO:tasks.rados.rados.0.smithi059.stdout:127: finishing write tid 4 to smithi05930959-39 2024-04-19T02:03:23.743 INFO:tasks.rados.rados.0.smithi059.stdout:130: copy_from oid 50 from oid 30 current snap is 6 2024-04-19T02:03:23.744 INFO:tasks.rados.rados.0.smithi059.stdout:131: delete oid 21 current snap is 6 2024-04-19T02:03:23.744 INFO:tasks.rados.rados.0.smithi059.stdout:127: finishing write tid 5 to smithi05930959-39 2024-04-19T02:03:23.744 INFO:tasks.rados.rados.0.smithi059.stdout:127: oid 39 updating version 0 to 22 2024-04-19T02:03:23.744 INFO:tasks.rados.rados.0.smithi059.stdout:127: oid 39 updating version 22 to 23 2024-04-19T02:03:23.745 INFO:tasks.rados.rados.0.smithi059.stdout:127: oid 39 version 23 is already newer than 21 2024-04-19T02:03:23.745 INFO:tasks.rados.rados.0.smithi059.stdout:127: oid 39 version 23 is already newer than 20 2024-04-19T02:03:23.745 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 39 v 23 (ObjNum 78 snap 6 seq_num 78) dirty exists 2024-04-19T02:03:23.745 INFO:tasks.rados.rados.0.smithi059.stdout:127: left oid 39 (ObjNum 78 snap 6 seq_num 78) 2024-04-19T02:03:23.745 INFO:tasks.rados.rados.0.smithi059.stdout:128: finishing write tid 1 to smithi05930959-27 2024-04-19T02:03:23.745 INFO:tasks.rados.rados.0.smithi059.stdout:128: finishing write tid 2 to smithi05930959-27 2024-04-19T02:03:23.749 INFO:tasks.rados.rados.0.smithi059.stdout:128: finishing write tid 3 to smithi05930959-27 2024-04-19T02:03:23.750 INFO:tasks.rados.rados.0.smithi059.stdout:128: finishing write tid 4 to smithi05930959-27 2024-04-19T02:03:23.750 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 29 v 25 (ObjNum 72 snap 6 seq_num 72) dirty exists 2024-04-19T02:03:23.751 INFO:tasks.rados.rados.0.smithi059.stdout:127: done (4 left) 2024-04-19T02:03:23.751 INFO:tasks.rados.rados.0.smithi059.stdout:129: done (3 left) 2024-04-19T02:03:23.751 INFO:tasks.rados.rados.0.smithi059.stdout:131: done (2 left) 2024-04-19T02:03:23.752 INFO:tasks.rados.rados.0.smithi059.stdout:132: snap_create 2024-04-19T02:03:23.752 INFO:tasks.rados.rados.0.smithi059.stdout:128: finishing write tid 5 to smithi05930959-27 2024-04-19T02:03:23.753 INFO:tasks.rados.rados.0.smithi059.stdout:128: oid 27 updating version 0 to 57 2024-04-19T02:03:23.753 INFO:tasks.rados.rados.0.smithi059.stdout:128: oid 27 version 57 is already newer than 55 2024-04-19T02:03:23.753 INFO:tasks.rados.rados.0.smithi059.stdout:128: oid 27 version 57 is already newer than 54 2024-04-19T02:03:23.753 INFO:tasks.rados.rados.0.smithi059.stdout:128: oid 27 version 57 is already newer than 56 2024-04-19T02:03:23.753 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 27 v 57 (ObjNum 79 snap 6 seq_num 79) dirty exists 2024-04-19T02:03:23.753 INFO:tasks.rados.rados.0.smithi059.stdout:128: left oid 27 (ObjNum 79 snap 6 seq_num 79) 2024-04-19T02:03:23.756 INFO:tasks.rados.rados.0.smithi059.stdout:130: finishing copy_from racing read to smithi05930959-50 2024-04-19T02:03:23.760 INFO:tasks.rados.rados.0.smithi059.stdout:130: finishing copy_from to smithi05930959-50 2024-04-19T02:03:23.760 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 50 v 11 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-19T02:03:23.771 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:23.771 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:23.782 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:03:23.783+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:03:23.786 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-19T02:03:23.884 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:23.887+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:23.947 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:23.948 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:23.963 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-19T02:03:23.985 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:23.983+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:24.086 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:24.087 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:24.087 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:24.087+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:24.100 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-19T02:03:24.187 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:24.187+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:24.223 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:24.224 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:24.238 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-19T02:03:24.278 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T02:03:24.288 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:24.287+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:24.389 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:24.391+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:24.405 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:24.405 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:24.419 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-19T02:03:24.438 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-04-19T02:03:24.438 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T02:03:24.456 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-04-19T02:03:24.456 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T02:03:24.474 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-04-19T02:03:24.474 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T02:03:24.489 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:24.487+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:24.494 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-04-19T02:03:24.494 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T02:03:24.516 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-04-19T02:03:24.516 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T02:03:24.538 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-04-19T02:03:24.539 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T02:03:24.562 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-04-19T02:03:24.562 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T02:03:24.586 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-04-19T02:03:24.586 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:24.586 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:24.586 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T02:03:24.591 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:24.591+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:24.600 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-19T02:03:24.610 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-04-19T02:03:24.610 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T02:03:24.634 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-04-19T02:03:24.634 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T02:03:24.657 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-04-19T02:03:24.657 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T02:03:24.680 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-04-19T02:03:24.681 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T02:03:24.691 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:24.691+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:24.715 INFO:tasks.rados.rados.0.smithi059.stdout:128: done (2 left) 2024-04-19T02:03:24.715 INFO:tasks.rados.rados.0.smithi059.stdout:130: done (1 left) 2024-04-19T02:03:24.715 INFO:tasks.rados.rados.0.smithi059.stdout:132: done (0 left) 2024-04-19T02:03:24.715 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 44 current snap is 7 2024-04-19T02:03:24.715 INFO:tasks.rados.rados.0.smithi059.stdout:133: seq_num 81 ranges {737280=606208,1343488=720896,2064384=49152} 2024-04-19T02:03:24.723 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:24.724 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:03:24.724 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-04-19T02:03:24.724 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:24.730 INFO:tasks.rados.rados.0.smithi059.stdout:133: writing smithi05930959-44 from 737280 to 1343488 tid 1 2024-04-19T02:03:24.740 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-19T02:03:24.740 INFO:tasks.rados.rados.0.smithi059.stdout:133: writing smithi05930959-44 from 1343488 to 2064384 tid 2 2024-04-19T02:03:24.741 INFO:tasks.rados.rados.0.smithi059.stdout:133: writing smithi05930959-44 from 2064384 to 2113536 tid 3 2024-04-19T02:03:24.741 INFO:tasks.rados.rados.0.smithi059.stdout:134: copy_from oid 10 from oid 13 current snap is 7 2024-04-19T02:03:24.741 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 34 current snap is 7 2024-04-19T02:03:24.741 INFO:tasks.rados.rados.0.smithi059.stdout:135: seq_num 82 ranges {475136=622592,1097728=475136,1572864=294912} 2024-04-19T02:03:24.756 INFO:tasks.rados.rados.0.smithi059.stdout:135: writing smithi05930959-34 from 475136 to 1097728 tid 1 2024-04-19T02:03:24.762 INFO:tasks.rados.rados.0.smithi059.stdout:135: writing smithi05930959-34 from 1097728 to 1572864 tid 2 2024-04-19T02:03:24.771 INFO:tasks.rados.rados.0.smithi059.stdout:135: writing smithi05930959-34 from 1572864 to 1867776 tid 3 2024-04-19T02:03:24.771 INFO:tasks.rados.rados.0.smithi059.stdout:136: read oid 27 snap -1 2024-04-19T02:03:24.771 INFO:tasks.rados.rados.0.smithi059.stdout:136: expect (ObjNum 79 snap 6 seq_num 79) 2024-04-19T02:03:24.771 INFO:tasks.rados.rados.0.smithi059.stdout:137: copy_from oid 42 from oid 36 current snap is 7 2024-04-19T02:03:24.771 INFO:tasks.rados.rados.0.smithi059.stdout:133: finishing write tid 1 to smithi05930959-44 2024-04-19T02:03:24.771 INFO:tasks.rados.rados.0.smithi059.stdout:133: finishing write tid 2 to smithi05930959-44 2024-04-19T02:03:24.771 INFO:tasks.rados.rados.0.smithi059.stdout:133: finishing write tid 3 to smithi05930959-44 2024-04-19T02:03:24.771 INFO:tasks.rados.rados.0.smithi059.stdout:133: finishing write tid 4 to smithi05930959-44 2024-04-19T02:03:24.772 INFO:tasks.rados.rados.0.smithi059.stdout:133: finishing write tid 5 to smithi05930959-44 2024-04-19T02:03:24.772 INFO:tasks.rados.rados.0.smithi059.stdout:133: oid 44 updating version 0 to 28 2024-04-19T02:03:24.772 INFO:tasks.rados.rados.0.smithi059.stdout:133: oid 44 version 28 is already newer than 27 2024-04-19T02:03:24.772 INFO:tasks.rados.rados.0.smithi059.stdout:133: oid 44 version 28 is already newer than 26 2024-04-19T02:03:24.772 INFO:tasks.rados.rados.0.smithi059.stdout:133: oid 44 version 28 is already newer than 25 2024-04-19T02:03:24.772 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 44 v 28 (ObjNum 81 snap 7 seq_num 81) dirty exists 2024-04-19T02:03:24.772 INFO:tasks.rados.rados.0.smithi059.stdout:133: left oid 44 (ObjNum 81 snap 7 seq_num 81) 2024-04-19T02:03:24.772 INFO:tasks.rados.rados.0.smithi059.stdout:134: finishing copy_from racing read to smithi05930959-10 2024-04-19T02:03:24.772 INFO:tasks.rados.rados.0.smithi059.stdout:133: done (4 left) 2024-04-19T02:03:24.772 INFO:tasks.rados.rados.0.smithi059.stdout:138: snap_create 2024-04-19T02:03:24.772 INFO:tasks.rados.rados.0.smithi059.stdout:135: finishing write tid 1 to smithi05930959-34 2024-04-19T02:03:24.776 INFO:tasks.rados.rados.0.smithi059.stdout:134: finishing copy_from to smithi05930959-10 2024-04-19T02:03:24.777 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 10 v 11 (ObjNum 56 snap 1 seq_num 56) dirty exists 2024-04-19T02:03:24.777 INFO:tasks.rados.rados.0.smithi059.stdout:135: finishing write tid 2 to smithi05930959-34 2024-04-19T02:03:24.777 INFO:tasks.rados.rados.0.smithi059.stdout:135: finishing write tid 3 to smithi05930959-34 2024-04-19T02:03:24.778 INFO:tasks.rados.rados.0.smithi059.stdout:136: expect (ObjNum 79 snap 6 seq_num 79) 2024-04-19T02:03:24.792 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:24.791+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:24.815 INFO:tasks.rados.rados.0.smithi059.stdout:135: finishing write tid 4 to smithi05930959-34 2024-04-19T02:03:24.815 INFO:tasks.rados.rados.0.smithi059.stdout:135: finishing write tid 5 to smithi05930959-34 2024-04-19T02:03:24.815 INFO:tasks.rados.rados.0.smithi059.stdout:135: oid 34 updating version 0 to 22 2024-04-19T02:03:24.816 INFO:tasks.rados.rados.0.smithi059.stdout:135: oid 34 version 22 is already newer than 21 2024-04-19T02:03:24.816 INFO:tasks.rados.rados.0.smithi059.stdout:135: oid 34 updating version 22 to 23 2024-04-19T02:03:24.816 INFO:tasks.rados.rados.0.smithi059.stdout:135: oid 34 version 23 is already newer than 20 2024-04-19T02:03:24.816 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 34 v 23 (ObjNum 82 snap 7 seq_num 82) dirty exists 2024-04-19T02:03:24.816 INFO:tasks.rados.rados.0.smithi059.stdout:135: left oid 34 (ObjNum 82 snap 7 seq_num 82) 2024-04-19T02:03:24.816 INFO:tasks.rados.rados.0.smithi059.stdout:137: finishing copy_from racing read to smithi05930959-42 2024-04-19T02:03:24.816 INFO:tasks.rados.rados.0.smithi059.stdout:137: finishing copy_from to smithi05930959-42 2024-04-19T02:03:24.816 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 42 v 6 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-19T02:03:24.869 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:24.869 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:03:24.869 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-04-19T02:03:24.869 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-04-19T02:03:24.869 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:24.883 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-19T02:03:24.893 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:24.890+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:24.994 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:24.995+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:25.057 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:25.057 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-04-19T02:03:25.057 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-04-19T02:03:25.057 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-04-19T02:03:25.058 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.058 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "pg_update_log_missing(3.es4 epoch 32/26 rep_tid 18 entries 32'12 (0'0) error 3:79fe7502:::smithi05930959-12:head by client.4509.0:240 0.000000 -2 [r=-2+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 30'10)", 2024-04-19T02:03:25.058 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.651416+0000", 2024-04-19T02:03:25.058 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 8.4070604299999996, 2024-04-19T02:03:25.058 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0021345940000000001, 2024-04-19T02:03:25.058 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.058 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.058 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.058 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.058 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.058 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.059 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.651416+0000", 2024-04-19T02:03:25.059 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.059 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.059 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.059 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.059 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.651416+0000", 2024-04-19T02:03:25.059 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.059 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.059 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.059 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.059 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.651417+0000", 2024-04-19T02:03:25.059 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.6300000000000005e-07 2024-04-19T02:03:25.059 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.059 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.060 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.060 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.651435+0000", 2024-04-19T02:03:25.060 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8156e-05 2024-04-19T02:03:25.060 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.060 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.060 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.060 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.651492+0000", 2024-04-19T02:03:25.060 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.6737999999999998e-05 2024-04-19T02:03:25.060 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.060 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.060 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.060 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.651531+0000", 2024-04-19T02:03:25.060 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9675999999999998e-05 2024-04-19T02:03:25.060 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.060 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.061 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.061 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.651818+0000", 2024-04-19T02:03:25.061 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00028672799999999998 2024-04-19T02:03:25.061 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.061 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.061 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.061 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.653551+0000", 2024-04-19T02:03:25.061 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0017326329999999999 2024-04-19T02:03:25.061 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.061 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.061 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.061 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.061 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.061 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s7 35/26 ECSubWrite(tid=29, reqid=osd.1.0:29, at_version=35'18, trim_to=0'0, roll_forward_to=34'16))", 2024-04-19T02:03:25.062 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.667280+0000", 2024-04-19T02:03:25.062 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 5.3911970279999997, 2024-04-19T02:03:25.062 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002716478, 2024-04-19T02:03:25.062 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.062 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.062 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.062 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.062 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.062 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.062 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.667280+0000", 2024-04-19T02:03:25.062 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.062 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.062 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.062 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.063 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.667280+0000", 2024-04-19T02:03:25.063 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.063 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.063 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.063 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.063 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.667280+0000", 2024-04-19T02:03:25.063 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2000000000000001e-07 2024-04-19T02:03:25.063 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.063 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.063 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.063 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.667304+0000", 2024-04-19T02:03:25.063 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4249000000000001e-05 2024-04-19T02:03:25.063 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.063 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.063 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.064 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.667321+0000", 2024-04-19T02:03:25.064 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6889e-05 2024-04-19T02:03:25.064 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.064 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.064 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.064 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.667330+0000", 2024-04-19T02:03:25.064 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3719999999999994e-06 2024-04-19T02:03:25.064 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.064 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.064 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.064 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.667537+0000", 2024-04-19T02:03:25.064 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00020656200000000001 2024-04-19T02:03:25.064 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.064 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.064 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.065 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.667560+0000", 2024-04-19T02:03:25.065 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2860000000000001e-05 2024-04-19T02:03:25.065 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.065 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.065 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.065 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.669957+0000", 2024-04-19T02:03:25.065 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0023968819999999999 2024-04-19T02:03:25.065 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.065 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.065 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.065 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.669996+0000", 2024-04-19T02:03:25.065 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9443999999999997e-05 2024-04-19T02:03:25.065 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.065 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.065 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.066 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.066 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.066 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.3s2 35/26 ECSubWrite(tid=29, reqid=osd.10.0:29, at_version=35'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-19T02:03:25.066 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.671172+0000", 2024-04-19T02:03:25.066 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 5.387304565, 2024-04-19T02:03:25.066 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0015971539999999999, 2024-04-19T02:03:25.066 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.066 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.066 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.066 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.066 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.066 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.066 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.671172+0000", 2024-04-19T02:03:25.066 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.067 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.067 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.067 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.067 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.671172+0000", 2024-04-19T02:03:25.067 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.067 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.067 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.067 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.067 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.671172+0000", 2024-04-19T02:03:25.067 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7399999999999999e-07 2024-04-19T02:03:25.067 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.067 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.067 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.068 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.671193+0000", 2024-04-19T02:03:25.068 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0699e-05 2024-04-19T02:03:25.068 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.068 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.068 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.068 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.671205+0000", 2024-04-19T02:03:25.068 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2122e-05 2024-04-19T02:03:25.068 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.068 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.068 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.068 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.671214+0000", 2024-04-19T02:03:25.068 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.4099999999999997e-06 2024-04-19T02:03:25.068 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.068 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.069 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.069 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.671281+0000", 2024-04-19T02:03:25.069 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.6451000000000005e-05 2024-04-19T02:03:25.069 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.069 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.069 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.069 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.671292+0000", 2024-04-19T02:03:25.069 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0718e-05 2024-04-19T02:03:25.069 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.069 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.069 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.069 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.672731+0000", 2024-04-19T02:03:25.069 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001439334 2024-04-19T02:03:25.069 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.069 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.070 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.070 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.672769+0000", 2024-04-19T02:03:25.070 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.8145999999999997e-05 2024-04-19T02:03:25.070 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.070 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.070 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.070 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.070 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.070 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.7s3 35/26 ECSubWrite(tid=94, reqid=osd.3.0:93, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-04-19T02:03:25.070 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.673828+0000", 2024-04-19T02:03:25.070 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 5.3846484090000004, 2024-04-19T02:03:25.070 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0031614579999999998, 2024-04-19T02:03:25.070 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.070 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.070 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.071 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.071 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.071 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.071 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.673828+0000", 2024-04-19T02:03:25.071 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.071 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.071 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.071 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.071 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.673828+0000", 2024-04-19T02:03:25.071 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.071 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.071 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.071 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.071 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.673829+0000", 2024-04-19T02:03:25.072 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9000000000000002e-07 2024-04-19T02:03:25.072 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.072 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.072 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.072 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.673838+0000", 2024-04-19T02:03:25.072 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.1249999999999999e-06 2024-04-19T02:03:25.072 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.072 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.072 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.072 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.673846+0000", 2024-04-19T02:03:25.073 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7139999999999999e-06 2024-04-19T02:03:25.073 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.073 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.073 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.073 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.673855+0000", 2024-04-19T02:03:25.073 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.6880000000000005e-06 2024-04-19T02:03:25.073 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.073 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.073 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.073 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.674115+0000", 2024-04-19T02:03:25.073 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00025984100000000002 2024-04-19T02:03:25.073 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.073 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.073 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.073 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.674124+0000", 2024-04-19T02:03:25.074 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.4930000000000004e-06 2024-04-19T02:03:25.074 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.074 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.074 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.074 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.676938+0000", 2024-04-19T02:03:25.074 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002813903 2024-04-19T02:03:25.074 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.074 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.074 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.074 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.676990+0000", 2024-04-19T02:03:25.074 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.1304000000000002e-05 2024-04-19T02:03:25.074 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.074 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.074 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.074 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.075 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.075 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es4 35/26 ECSubWrite(tid=23, reqid=osd.7.0:23, at_version=35'18, trim_to=0'0, roll_forward_to=34'16))", 2024-04-19T02:03:25.075 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.676071+0000", 2024-04-19T02:03:25.075 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 5.3824051490000002, 2024-04-19T02:03:25.075 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0084307070000000008, 2024-04-19T02:03:25.075 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.075 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.075 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.075 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.075 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.075 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.075 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.676071+0000", 2024-04-19T02:03:25.075 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.075 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.075 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.076 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.076 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.676071+0000", 2024-04-19T02:03:25.076 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.076 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.076 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.076 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.076 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.676072+0000", 2024-04-19T02:03:25.076 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7000000000000001e-07 2024-04-19T02:03:25.076 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.076 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.076 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.077 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.676098+0000", 2024-04-19T02:03:25.077 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6169999999999998e-05 2024-04-19T02:03:25.077 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.077 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.077 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.077 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.676110+0000", 2024-04-19T02:03:25.077 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2568000000000001e-05 2024-04-19T02:03:25.077 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.077 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.077 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.077 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.676122+0000", 2024-04-19T02:03:25.077 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1371e-05 2024-04-19T02:03:25.077 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.077 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.077 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.078 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.676504+0000", 2024-04-19T02:03:25.078 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00038213599999999999 2024-04-19T02:03:25.078 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.078 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.078 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.078 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.676513+0000", 2024-04-19T02:03:25.078 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.9509999999999995e-06 2024-04-19T02:03:25.078 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.078 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.078 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.078 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.684458+0000", 2024-04-19T02:03:25.078 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0079447419999999994 2024-04-19T02:03:25.078 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.078 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.078 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.079 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.684502+0000", 2024-04-19T02:03:25.079 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4499000000000002e-05 2024-04-19T02:03:25.079 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.079 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.079 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.079 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.079 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.079 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s7 35/26 ECSubWrite(tid=30, reqid=osd.1.0:29, at_version=0'0, trim_to=0'0, roll_forward_to=35'18))", 2024-04-19T02:03:25.079 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.680702+0000", 2024-04-19T02:03:25.079 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 5.3777744949999997, 2024-04-19T02:03:25.079 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0072111470000000002, 2024-04-19T02:03:25.079 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.079 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.079 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.080 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.080 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.080 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.080 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680702+0000", 2024-04-19T02:03:25.080 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.080 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.080 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.080 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.080 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680702+0000", 2024-04-19T02:03:25.080 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.080 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.080 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.080 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.081 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680703+0000", 2024-04-19T02:03:25.081 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3099999999999996e-07 2024-04-19T02:03:25.081 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.081 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.081 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.081 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680717+0000", 2024-04-19T02:03:25.081 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4515000000000001e-05 2024-04-19T02:03:25.081 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.081 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.081 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.081 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680737+0000", 2024-04-19T02:03:25.081 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.984e-05 2024-04-19T02:03:25.081 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.081 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.082 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.082 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680747+0000", 2024-04-19T02:03:25.082 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.7410000000000004e-06 2024-04-19T02:03:25.082 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.082 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.082 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.082 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680846+0000", 2024-04-19T02:03:25.082 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.8572000000000004e-05 2024-04-19T02:03:25.082 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.082 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.082 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.082 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680857+0000", 2024-04-19T02:03:25.082 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1055e-05 2024-04-19T02:03:25.082 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.082 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.083 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.083 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.687866+0000", 2024-04-19T02:03:25.083 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0070097149999999997 2024-04-19T02:03:25.083 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.083 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.083 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.083 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.687913+0000", 2024-04-19T02:03:25.083 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.6878000000000001e-05 2024-04-19T02:03:25.083 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.083 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.083 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.083 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.083 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.083 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs6 35/26 ECSubWrite(tid=54, reqid=client.4509.0:271, at_version=35'14, trim_to=0'0, roll_forward_to=34'13))", 2024-04-19T02:03:25.083 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.684579+0000", 2024-04-19T02:03:25.084 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 5.3738979410000001, 2024-04-19T02:03:25.084 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.003879979, 2024-04-19T02:03:25.084 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.084 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.084 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.084 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.084 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.084 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.084 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.684579+0000", 2024-04-19T02:03:25.084 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.084 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.084 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.084 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.084 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.684579+0000", 2024-04-19T02:03:25.084 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.085 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.085 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.085 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.085 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.684579+0000", 2024-04-19T02:03:25.085 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3099999999999999e-07 2024-04-19T02:03:25.085 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.085 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.085 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.086 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.685043+0000", 2024-04-19T02:03:25.086 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000464344 2024-04-19T02:03:25.086 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.086 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.086 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.086 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.685058+0000", 2024-04-19T02:03:25.086 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.468e-05 2024-04-19T02:03:25.086 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.086 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.086 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.086 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.685067+0000", 2024-04-19T02:03:25.086 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.1400000000000006e-06 2024-04-19T02:03:25.086 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.086 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.086 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.087 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.685168+0000", 2024-04-19T02:03:25.087 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000100691 2024-04-19T02:03:25.087 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.087 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.087 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.087 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.685183+0000", 2024-04-19T02:03:25.087 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.522e-05 2024-04-19T02:03:25.087 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.087 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.087 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.087 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.688412+0000", 2024-04-19T02:03:25.087 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.003229375 2024-04-19T02:03:25.087 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.087 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.088 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.088 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.688459+0000", 2024-04-19T02:03:25.088 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.6298000000000003e-05 2024-04-19T02:03:25.088 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.088 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.088 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.088 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.088 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.088 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.3s2 35/26 ECSubWrite(tid=30, reqid=osd.10.0:29, at_version=0'0, trim_to=0'0, roll_forward_to=35'8))", 2024-04-19T02:03:25.088 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.691833+0000", 2024-04-19T02:03:25.088 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 5.366643732, 2024-04-19T02:03:25.088 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0014376129999999999, 2024-04-19T02:03:25.088 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.088 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.088 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.089 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.089 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.089 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.089 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.691833+0000", 2024-04-19T02:03:25.089 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.089 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.089 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.089 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.089 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.691833+0000", 2024-04-19T02:03:25.089 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.089 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.089 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.089 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.089 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.691834+0000", 2024-04-19T02:03:25.090 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3699999999999999e-07 2024-04-19T02:03:25.090 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.090 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.090 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.090 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.691848+0000", 2024-04-19T02:03:25.090 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4654000000000001e-05 2024-04-19T02:03:25.090 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.090 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.090 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.090 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.691870+0000", 2024-04-19T02:03:25.090 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1316000000000001e-05 2024-04-19T02:03:25.090 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.090 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.090 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.091 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.691880+0000", 2024-04-19T02:03:25.091 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0172e-05 2024-04-19T02:03:25.091 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.091 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.091 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.091 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.691957+0000", 2024-04-19T02:03:25.091 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6803000000000003e-05 2024-04-19T02:03:25.091 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.091 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.091 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.091 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.691967+0000", 2024-04-19T02:03:25.091 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0631e-05 2024-04-19T02:03:25.091 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.091 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.092 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.092 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.693225+0000", 2024-04-19T02:03:25.092 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0012572219999999999 2024-04-19T02:03:25.092 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.092 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.092 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.092 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.693270+0000", 2024-04-19T02:03:25.092 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5877999999999997e-05 2024-04-19T02:03:25.092 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.092 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.092 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.092 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.092 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.093 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs6 35/26 ECSubWrite(tid=57, reqid=client.4509.0:273, at_version=0'0, trim_to=0'0, roll_forward_to=35'16))", 2024-04-19T02:03:25.093 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.702063+0000", 2024-04-19T02:03:25.093 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 5.3564134599999997, 2024-04-19T02:03:25.093 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0020864199999999999, 2024-04-19T02:03:25.093 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.093 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.093 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.093 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.093 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.093 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.093 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.702063+0000", 2024-04-19T02:03:25.093 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.093 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.093 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.093 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.093 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.702063+0000", 2024-04-19T02:03:25.094 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.094 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.094 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.094 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.094 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.702064+0000", 2024-04-19T02:03:25.094 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0270000000000001e-06 2024-04-19T02:03:25.094 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.094 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.094 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.094 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.702077+0000", 2024-04-19T02:03:25.094 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3315999999999999e-05 2024-04-19T02:03:25.094 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.094 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.094 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.095 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.702097+0000", 2024-04-19T02:03:25.095 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9936999999999999e-05 2024-04-19T02:03:25.095 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.095 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.095 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.095 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.702108+0000", 2024-04-19T02:03:25.095 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0516000000000001e-05 2024-04-19T02:03:25.095 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.095 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.095 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.095 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.702209+0000", 2024-04-19T02:03:25.095 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000100772 2024-04-19T02:03:25.095 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.095 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.095 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.096 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.702220+0000", 2024-04-19T02:03:25.096 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0932e-05 2024-04-19T02:03:25.096 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.096 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.096 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.096 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.704102+0000", 2024-04-19T02:03:25.096 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0018825529999999999 2024-04-19T02:03:25.096 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.096 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.096 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.096 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.704150+0000", 2024-04-19T02:03:25.096 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.7367000000000002e-05 2024-04-19T02:03:25.096 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.096 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.096 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.097 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.097 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.097 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.3s2 35/26 ECSubWrite(tid=36, reqid=client.4509.0:289, at_version=35'9, trim_to=0'0, roll_forward_to=35'8))", 2024-04-19T02:03:25.097 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.786746+0000", 2024-04-19T02:03:25.097 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 5.2717304409999999, 2024-04-19T02:03:25.097 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0014080900000000001, 2024-04-19T02:03:25.097 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.097 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.097 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.097 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.097 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.097 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.097 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.786746+0000", 2024-04-19T02:03:25.097 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.097 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.098 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.098 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.098 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.786746+0000", 2024-04-19T02:03:25.098 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.098 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.098 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.098 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.098 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.786747+0000", 2024-04-19T02:03:25.098 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.5199999999999996e-07 2024-04-19T02:03:25.098 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.098 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.098 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.098 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.786981+0000", 2024-04-19T02:03:25.098 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000234267 2024-04-19T02:03:25.098 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.099 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.099 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.099 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.787000+0000", 2024-04-19T02:03:25.099 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9009999999999999e-05 2024-04-19T02:03:25.099 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.099 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.099 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.099 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.787010+0000", 2024-04-19T02:03:25.099 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.798e-06 2024-04-19T02:03:25.099 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.099 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.099 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.099 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.787080+0000", 2024-04-19T02:03:25.099 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.0469000000000006e-05 2024-04-19T02:03:25.100 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.100 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.100 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.100 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.787092+0000", 2024-04-19T02:03:25.100 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.134e-05 2024-04-19T02:03:25.100 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.100 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.100 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.100 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.788118+0000", 2024-04-19T02:03:25.100 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001026117 2024-04-19T02:03:25.100 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.100 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.100 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.100 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.788154+0000", 2024-04-19T02:03:25.100 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6337000000000001e-05 2024-04-19T02:03:25.101 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.101 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.101 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.101 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.101 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.101 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s7 40/26 ECSubWrite(tid=37, reqid=client.4509.0:344, at_version=40'25, trim_to=0'0, roll_forward_to=39'23))", 2024-04-19T02:03:25.101 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.739333+0000", 2024-04-19T02:03:25.101 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.31914331800000001, 2024-04-19T02:03:25.101 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0023397930000000002, 2024-04-19T02:03:25.101 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.101 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.101 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.101 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.101 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.101 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.102 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.739333+0000", 2024-04-19T02:03:25.102 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.102 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.102 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.102 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.102 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.739333+0000", 2024-04-19T02:03:25.102 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.102 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.102 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.102 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.102 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.739334+0000", 2024-04-19T02:03:25.102 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4799999999999999e-07 2024-04-19T02:03:25.102 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.102 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.103 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.103 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.739490+0000", 2024-04-19T02:03:25.103 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000156775 2024-04-19T02:03:25.103 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.103 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.103 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.103 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.739514+0000", 2024-04-19T02:03:25.103 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3189000000000001e-05 2024-04-19T02:03:25.103 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.103 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.103 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.103 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.739526+0000", 2024-04-19T02:03:25.103 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2345e-05 2024-04-19T02:03:25.103 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.104 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.104 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.104 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.739641+0000", 2024-04-19T02:03:25.104 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000115186 2024-04-19T02:03:25.104 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.104 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.104 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.104 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.739656+0000", 2024-04-19T02:03:25.104 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5084e-05 2024-04-19T02:03:25.104 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.104 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.104 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.105 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.741627+0000", 2024-04-19T02:03:25.105 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0019712979999999998 2024-04-19T02:03:25.105 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.105 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.105 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.105 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.741673+0000", 2024-04-19T02:03:25.105 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5568000000000001e-05 2024-04-19T02:03:25.105 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.105 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.105 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.105 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.105 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.105 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s7 40/26 ECSubWrite(tid=38, reqid=client.4509.0:345, at_version=40'26, trim_to=0'0, roll_forward_to=39'23))", 2024-04-19T02:03:25.105 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.747573+0000", 2024-04-19T02:03:25.106 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.31090385700000001, 2024-04-19T02:03:25.106 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0035957490000000001, 2024-04-19T02:03:25.106 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.106 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.106 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.106 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.106 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.106 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.106 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.747573+0000", 2024-04-19T02:03:25.106 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.106 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.106 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.106 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.106 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.747573+0000", 2024-04-19T02:03:25.107 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.107 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.107 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.107 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.107 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.747574+0000", 2024-04-19T02:03:25.107 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5269999999999999e-06 2024-04-19T02:03:25.107 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.107 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.107 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.107 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.747756+0000", 2024-04-19T02:03:25.107 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00018225899999999999 2024-04-19T02:03:25.107 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.107 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.107 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.107 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.747777+0000", 2024-04-19T02:03:25.108 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0818e-05 2024-04-19T02:03:25.108 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.108 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.108 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.108 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.747790+0000", 2024-04-19T02:03:25.108 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2224e-05 2024-04-19T02:03:25.108 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.108 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.108 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.108 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.747905+0000", 2024-04-19T02:03:25.108 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00011573700000000001 2024-04-19T02:03:25.108 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.108 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.108 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.108 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.747919+0000", 2024-04-19T02:03:25.108 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4076999999999999e-05 2024-04-19T02:03:25.109 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.109 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.109 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.109 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.751120+0000", 2024-04-19T02:03:25.109 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0032010950000000002 2024-04-19T02:03:25.109 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.109 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.109 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.109 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.751168+0000", 2024-04-19T02:03:25.109 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.8012e-05 2024-04-19T02:03:25.109 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.109 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.109 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.109 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.110 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.110 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s7 40/26 ECSubWrite(tid=39, reqid=client.4509.0:346, at_version=40'27, trim_to=0'0, roll_forward_to=39'23))", 2024-04-19T02:03:25.110 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.748951+0000", 2024-04-19T02:03:25.110 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.30952549000000001, 2024-04-19T02:03:25.110 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0029204449999999998, 2024-04-19T02:03:25.110 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.110 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.110 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.110 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.110 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.110 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.110 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.748951+0000", 2024-04-19T02:03:25.110 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.110 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.110 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.111 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.111 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.748951+0000", 2024-04-19T02:03:25.111 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.111 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.111 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.111 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.111 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.748952+0000", 2024-04-19T02:03:25.111 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7599999999999996e-07 2024-04-19T02:03:25.111 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.111 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.111 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.111 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.748986+0000", 2024-04-19T02:03:25.111 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3939000000000002e-05 2024-04-19T02:03:25.111 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.111 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.112 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.112 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749006+0000", 2024-04-19T02:03:25.112 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0000000000000002e-05 2024-04-19T02:03:25.112 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.112 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.112 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.112 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749016+0000", 2024-04-19T02:03:25.112 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0186999999999999e-05 2024-04-19T02:03:25.112 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.112 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.112 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.112 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749113+0000", 2024-04-19T02:03:25.112 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.713e-05 2024-04-19T02:03:25.112 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.112 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.113 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.113 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749124+0000", 2024-04-19T02:03:25.113 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0818999999999999e-05 2024-04-19T02:03:25.113 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.113 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.113 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.113 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.751809+0000", 2024-04-19T02:03:25.113 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0026846309999999998 2024-04-19T02:03:25.113 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.113 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.113 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.113 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.751872+0000", 2024-04-19T02:03:25.113 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.2862999999999998e-05 2024-04-19T02:03:25.113 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.113 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.114 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.114 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.114 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.114 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s7 40/26 ECSubWrite(tid=40, reqid=client.4509.0:347, at_version=40'28, trim_to=0'0, roll_forward_to=40'25))", 2024-04-19T02:03:25.114 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.750019+0000", 2024-04-19T02:03:25.114 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.30845748499999998, 2024-04-19T02:03:25.114 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0018802109999999999, 2024-04-19T02:03:25.114 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.114 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.114 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.114 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.114 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.114 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.114 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750019+0000", 2024-04-19T02:03:25.115 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.115 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.115 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.115 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.115 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750019+0000", 2024-04-19T02:03:25.115 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.115 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.115 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.115 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.115 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750020+0000", 2024-04-19T02:03:25.115 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3200000000000003e-07 2024-04-19T02:03:25.115 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.115 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.115 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.115 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750038+0000", 2024-04-19T02:03:25.116 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8270999999999998e-05 2024-04-19T02:03:25.116 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.116 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.116 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.116 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750055+0000", 2024-04-19T02:03:25.116 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6833e-05 2024-04-19T02:03:25.116 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.116 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.116 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.116 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750064+0000", 2024-04-19T02:03:25.116 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3980000000000005e-06 2024-04-19T02:03:25.116 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.116 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.116 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.116 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750170+0000", 2024-04-19T02:03:25.117 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000105636 2024-04-19T02:03:25.117 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.117 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.117 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.117 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750180+0000", 2024-04-19T02:03:25.117 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.8649999999999996e-06 2024-04-19T02:03:25.117 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.117 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.117 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.117 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.751875+0000", 2024-04-19T02:03:25.117 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0016952880000000001 2024-04-19T02:03:25.117 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.117 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.117 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.117 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.751899+0000", 2024-04-19T02:03:25.118 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3988000000000001e-05 2024-04-19T02:03:25.118 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.118 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.118 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.118 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.118 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.118 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es4 40/26 ECSubWrite(tid=26, reqid=client.4509.0:351, at_version=40'20, trim_to=0'0, roll_forward_to=35'18))", 2024-04-19T02:03:25.118 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.765105+0000", 2024-04-19T02:03:25.118 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.29337160000000001, 2024-04-19T02:03:25.118 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0028039129999999999, 2024-04-19T02:03:25.118 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.118 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.118 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.118 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.118 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.119 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.119 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.765105+0000", 2024-04-19T02:03:25.119 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.119 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.119 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.119 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.119 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.765105+0000", 2024-04-19T02:03:25.119 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.119 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.119 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.119 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.119 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.765105+0000", 2024-04-19T02:03:25.119 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.0100000000000005e-07 2024-04-19T02:03:25.119 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.119 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.120 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.120 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.765565+0000", 2024-04-19T02:03:25.120 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00045951500000000002 2024-04-19T02:03:25.120 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.120 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.120 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.120 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.765582+0000", 2024-04-19T02:03:25.120 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7362000000000001e-05 2024-04-19T02:03:25.120 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.120 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.120 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.120 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.765599+0000", 2024-04-19T02:03:25.120 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6416999999999999e-05 2024-04-19T02:03:25.120 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.120 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.121 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.121 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.765687+0000", 2024-04-19T02:03:25.121 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.8141999999999997e-05 2024-04-19T02:03:25.121 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.121 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.121 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.121 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.765795+0000", 2024-04-19T02:03:25.121 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010825800000000001 2024-04-19T02:03:25.121 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.121 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.121 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.121 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.767862+0000", 2024-04-19T02:03:25.121 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0020663809999999999 2024-04-19T02:03:25.121 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.121 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.122 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.122 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.767909+0000", 2024-04-19T02:03:25.122 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.7336999999999998e-05 2024-04-19T02:03:25.122 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.122 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.122 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.122 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.122 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.122 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es4 40/26 ECSubWrite(tid=27, reqid=client.4509.0:352, at_version=40'21, trim_to=0'0, roll_forward_to=35'18))", 2024-04-19T02:03:25.122 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.771850+0000", 2024-04-19T02:03:25.122 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.28662643799999998, 2024-04-19T02:03:25.122 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0019278120000000001, 2024-04-19T02:03:25.122 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.122 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.122 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.123 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.123 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.123 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.123 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.771850+0000", 2024-04-19T02:03:25.123 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.123 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.123 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.123 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.123 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.771850+0000", 2024-04-19T02:03:25.123 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.123 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.123 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.123 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.123 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.771857+0000", 2024-04-19T02:03:25.123 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.4069999999999998e-06 2024-04-19T02:03:25.123 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.124 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.124 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.124 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772089+0000", 2024-04-19T02:03:25.124 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000232229 2024-04-19T02:03:25.124 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.124 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.124 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.124 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772116+0000", 2024-04-19T02:03:25.124 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7376999999999999e-05 2024-04-19T02:03:25.124 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.124 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.124 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.124 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772129+0000", 2024-04-19T02:03:25.124 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2568999999999999e-05 2024-04-19T02:03:25.124 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.125 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.125 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.125 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772204+0000", 2024-04-19T02:03:25.125 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.5480000000000002e-05 2024-04-19T02:03:25.125 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.125 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.125 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.125 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772216+0000", 2024-04-19T02:03:25.125 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1522000000000001e-05 2024-04-19T02:03:25.125 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.125 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.125 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.125 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.773736+0000", 2024-04-19T02:03:25.125 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0015204050000000001 2024-04-19T02:03:25.125 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.126 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.126 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.126 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.773778+0000", 2024-04-19T02:03:25.126 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1823000000000003e-05 2024-04-19T02:03:25.126 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.126 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.126 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.126 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.126 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.126 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(osd.9.18:2 3.1s0 3:874b3480:::smithi05930959-36:head [assert-version v6,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:25.126 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.775686+0000", 2024-04-19T02:03:25.126 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.28279068000000002, 2024-04-19T02:03:25.126 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010480552000000001, 2024-04-19T02:03:25.126 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.127 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.127 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:25.127 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.127 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.127 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.127 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.775686+0000", 2024-04-19T02:03:25.127 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.127 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.127 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.127 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.127 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.775686+0000", 2024-04-19T02:03:25.127 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.127 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.127 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.127 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.128 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.775687+0000", 2024-04-19T02:03:25.128 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3659999999999999e-06 2024-04-19T02:03:25.128 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.128 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.128 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.128 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.775694+0000", 2024-04-19T02:03:25.128 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.2540000000000003e-06 2024-04-19T02:03:25.128 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.128 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.128 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.128 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.775706+0000", 2024-04-19T02:03:25.128 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1654000000000001e-05 2024-04-19T02:03:25.128 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.128 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.128 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.129 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.775715+0000", 2024-04-19T02:03:25.129 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0189999999999995e-06 2024-04-19T02:03:25.129 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.129 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.129 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.129 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.775831+0000", 2024-04-19T02:03:25.129 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000115606 2024-04-19T02:03:25.129 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.129 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.129 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:25.129 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.776060+0000", 2024-04-19T02:03:25.129 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00022901700000000001 2024-04-19T02:03:25.129 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.129 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.130 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.130 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.786166+0000", 2024-04-19T02:03:25.130 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010106636 2024-04-19T02:03:25.130 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.130 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.130 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.130 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.130 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.130 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es4 40/26 ECSubWrite(tid=28, reqid=client.4509.0:353, at_version=40'22, trim_to=0'0, roll_forward_to=40'20))", 2024-04-19T02:03:25.130 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.776793+0000", 2024-04-19T02:03:25.130 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.28168345099999997, 2024-04-19T02:03:25.130 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001382903, 2024-04-19T02:03:25.130 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.130 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.130 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.131 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.131 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.131 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.131 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.776793+0000", 2024-04-19T02:03:25.131 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.131 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.131 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.131 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.131 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.776793+0000", 2024-04-19T02:03:25.131 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.131 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.131 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.131 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.131 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.776794+0000", 2024-04-19T02:03:25.131 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0759999999999999e-06 2024-04-19T02:03:25.132 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.132 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.132 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.132 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.776885+0000", 2024-04-19T02:03:25.132 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0470000000000006e-05 2024-04-19T02:03:25.132 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.132 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.132 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.132 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.776897+0000", 2024-04-19T02:03:25.132 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2388999999999999e-05 2024-04-19T02:03:25.132 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.132 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.132 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.132 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.776904+0000", 2024-04-19T02:03:25.132 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9700000000000002e-06 2024-04-19T02:03:25.133 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.133 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.133 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.133 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.776975+0000", 2024-04-19T02:03:25.133 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.0574000000000006e-05 2024-04-19T02:03:25.133 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.133 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.133 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.133 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.776994+0000", 2024-04-19T02:03:25.133 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9391000000000001e-05 2024-04-19T02:03:25.133 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.133 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.133 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.133 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.778141+0000", 2024-04-19T02:03:25.133 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0011474569999999999 2024-04-19T02:03:25.134 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.134 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.134 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.134 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.778176+0000", 2024-04-19T02:03:25.134 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4576000000000002e-05 2024-04-19T02:03:25.134 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.134 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.134 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.134 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.134 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.134 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es4 40/26 ECSubWrite(tid=29, reqid=client.4509.0:354, at_version=40'23, trim_to=0'0, roll_forward_to=40'20))", 2024-04-19T02:03:25.134 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.779563+0000", 2024-04-19T02:03:25.134 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.27891403999999997, 2024-04-19T02:03:25.134 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001371429, 2024-04-19T02:03:25.135 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.135 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.135 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.135 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.135 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.135 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.135 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.779563+0000", 2024-04-19T02:03:25.135 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.135 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.135 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.135 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.135 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.779563+0000", 2024-04-19T02:03:25.135 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.135 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.135 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.136 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.136 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.779564+0000", 2024-04-19T02:03:25.136 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.172e-06 2024-04-19T02:03:25.136 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.136 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.136 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.136 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.779590+0000", 2024-04-19T02:03:25.136 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6645e-05 2024-04-19T02:03:25.136 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.136 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.136 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.136 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.779612+0000", 2024-04-19T02:03:25.136 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1597e-05 2024-04-19T02:03:25.136 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.136 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.137 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.137 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.779624+0000", 2024-04-19T02:03:25.137 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1759e-05 2024-04-19T02:03:25.137 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.137 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.137 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.137 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.780068+0000", 2024-04-19T02:03:25.137 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00044421700000000001 2024-04-19T02:03:25.137 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.137 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.137 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.137 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.780079+0000", 2024-04-19T02:03:25.137 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0655999999999999e-05 2024-04-19T02:03:25.137 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.137 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.138 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.138 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.780892+0000", 2024-04-19T02:03:25.138 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00081319499999999996 2024-04-19T02:03:25.138 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.138 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.138 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.138 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.780934+0000", 2024-04-19T02:03:25.138 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2188000000000003e-05 2024-04-19T02:03:25.138 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.138 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.138 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.138 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.138 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.138 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs6 40/26 ECSubWrite(tid=30, reqid=client.4509.0:359, at_version=40'16, trim_to=0'0, roll_forward_to=39'14))", 2024-04-19T02:03:25.138 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.790559+0000", 2024-04-19T02:03:25.139 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.267917813, 2024-04-19T02:03:25.139 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002561986, 2024-04-19T02:03:25.139 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.139 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.139 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.139 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.139 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.139 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.139 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.790559+0000", 2024-04-19T02:03:25.139 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.139 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.139 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.139 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.139 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.790559+0000", 2024-04-19T02:03:25.139 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.140 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.140 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.140 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.140 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.790559+0000", 2024-04-19T02:03:25.140 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.6899999999999997e-07 2024-04-19T02:03:25.140 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.140 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.140 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.140 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.790969+0000", 2024-04-19T02:03:25.140 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00040956599999999999 2024-04-19T02:03:25.140 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.140 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.140 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.140 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.790985+0000", 2024-04-19T02:03:25.140 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5756999999999998e-05 2024-04-19T02:03:25.141 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.141 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.141 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.141 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.790998+0000", 2024-04-19T02:03:25.141 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3343000000000001e-05 2024-04-19T02:03:25.141 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.141 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.141 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.141 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.791203+0000", 2024-04-19T02:03:25.141 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00020488200000000001 2024-04-19T02:03:25.141 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.141 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.141 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.141 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.791229+0000", 2024-04-19T02:03:25.141 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6106999999999999e-05 2024-04-19T02:03:25.142 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.142 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.142 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.142 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.793090+0000", 2024-04-19T02:03:25.142 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001860474 2024-04-19T02:03:25.142 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.142 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.142 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.142 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.793121+0000", 2024-04-19T02:03:25.142 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1188e-05 2024-04-19T02:03:25.142 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.142 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.142 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.142 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.142 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.142 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:25.144 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-19T02:03:25.145 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:25.143+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:25.245 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:25.247+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:25.275 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:25.275 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:03:25.275 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-04-19T02:03:25.276 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:25.290 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-19T02:03:25.346 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:25.347+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:25.408 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:25.408 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:03:25.408 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-04-19T02:03:25.408 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-04-19T02:03:25.408 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:25.422 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-19T02:03:25.466 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:25.467+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:25.547 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:25.547 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-04-19T02:03:25.547 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-04-19T02:03:25.548 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-04-19T02:03:25.548 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.548 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.2s5 31/26 ECSubWrite(tid=65, reqid=client.4509.0:230, at_version=31'24, trim_to=0'0, roll_forward_to=30'22))", 2024-04-19T02:03:25.548 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:15.641554+0000", 2024-04-19T02:03:25.548 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 9.9059709060000003, 2024-04-19T02:03:25.548 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0044342060000000004, 2024-04-19T02:03:25.548 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.548 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.548 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.548 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.548 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.548 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.548 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.641554+0000", 2024-04-19T02:03:25.549 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.549 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.549 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.549 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.549 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.641554+0000", 2024-04-19T02:03:25.549 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.549 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.549 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.549 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.549 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.641554+0000", 2024-04-19T02:03:25.549 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7299999999999996e-07 2024-04-19T02:03:25.549 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.549 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.549 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.549 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.641614+0000", 2024-04-19T02:03:25.549 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.9466000000000003e-05 2024-04-19T02:03:25.550 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.550 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.550 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.550 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.641640+0000", 2024-04-19T02:03:25.550 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6576000000000001e-05 2024-04-19T02:03:25.550 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.550 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.550 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.550 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.641665+0000", 2024-04-19T02:03:25.550 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4454e-05 2024-04-19T02:03:25.550 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.550 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.550 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.550 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.641865+0000", 2024-04-19T02:03:25.551 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00020016200000000001 2024-04-19T02:03:25.551 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.551 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.551 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.551 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.641890+0000", 2024-04-19T02:03:25.551 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5539000000000001e-05 2024-04-19T02:03:25.551 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.551 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.551 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.551 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.645924+0000", 2024-04-19T02:03:25.551 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0040340059999999997 2024-04-19T02:03:25.551 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.551 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.551 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.551 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.645988+0000", 2024-04-19T02:03:25.552 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.3429999999999994e-05 2024-04-19T02:03:25.552 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.552 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.552 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.552 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.552 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.552 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.2s5 31/26 ECSubWrite(tid=67, reqid=client.4509.0:231, at_version=31'26, trim_to=0'0, roll_forward_to=31'24))", 2024-04-19T02:03:25.552 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:15.661119+0000", 2024-04-19T02:03:25.552 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 9.886406011, 2024-04-19T02:03:25.552 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00379956, 2024-04-19T02:03:25.552 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.552 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.552 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.552 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.552 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.553 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.553 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661119+0000", 2024-04-19T02:03:25.553 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.553 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.553 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.553 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.553 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661119+0000", 2024-04-19T02:03:25.553 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.553 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.553 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.553 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.553 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661121+0000", 2024-04-19T02:03:25.553 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9539999999999998e-06 2024-04-19T02:03:25.553 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.553 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.553 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.554 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661805+0000", 2024-04-19T02:03:25.554 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00068432400000000004 2024-04-19T02:03:25.554 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.554 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.554 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.554 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661852+0000", 2024-04-19T02:03:25.554 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.6792999999999997e-05 2024-04-19T02:03:25.554 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.554 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.554 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.555 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661928+0000", 2024-04-19T02:03:25.555 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6630000000000003e-05 2024-04-19T02:03:25.555 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.555 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.555 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.555 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.662125+0000", 2024-04-19T02:03:25.555 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00019659500000000001 2024-04-19T02:03:25.555 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.555 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.555 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.555 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.662152+0000", 2024-04-19T02:03:25.555 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7033e-05 2024-04-19T02:03:25.555 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.555 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.555 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.555 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.664830+0000", 2024-04-19T02:03:25.556 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0026782389999999998 2024-04-19T02:03:25.556 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.556 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.556 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.556 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.664918+0000", 2024-04-19T02:03:25.556 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7991999999999994e-05 2024-04-19T02:03:25.556 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.556 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.556 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.556 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.556 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.556 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.2s5 32/26 ECSubWrite(tid=72, reqid=client.4509.0:236, at_version=32'29, trim_to=0'0, roll_forward_to=31'28))", 2024-04-19T02:03:25.556 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.654673+0000", 2024-04-19T02:03:25.556 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 8.8928516119999994, 2024-04-19T02:03:25.556 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0048911400000000004, 2024-04-19T02:03:25.557 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.557 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.557 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.557 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.557 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.557 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.557 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.654673+0000", 2024-04-19T02:03:25.557 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.557 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.557 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.557 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.557 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.654673+0000", 2024-04-19T02:03:25.557 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.557 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.557 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.557 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.558 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.654674+0000", 2024-04-19T02:03:25.558 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.1699999999999997e-07 2024-04-19T02:03:25.558 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.558 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.558 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.558 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.654793+0000", 2024-04-19T02:03:25.558 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000119443 2024-04-19T02:03:25.558 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.558 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.558 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.558 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.654826+0000", 2024-04-19T02:03:25.558 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2359e-05 2024-04-19T02:03:25.558 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.558 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.559 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.559 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.654865+0000", 2024-04-19T02:03:25.559 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9128999999999998e-05 2024-04-19T02:03:25.559 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.559 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.559 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.559 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.655228+0000", 2024-04-19T02:03:25.559 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000363669 2024-04-19T02:03:25.559 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.559 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.559 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.559 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.655253+0000", 2024-04-19T02:03:25.559 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4355e-05 2024-04-19T02:03:25.559 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.559 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.560 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.560 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.659500+0000", 2024-04-19T02:03:25.560 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0042475869999999997 2024-04-19T02:03:25.560 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.560 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.560 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.560 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.659564+0000", 2024-04-19T02:03:25.560 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.3781000000000005e-05 2024-04-19T02:03:25.560 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.560 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.560 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.560 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.560 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.560 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.2s5 32/26 ECSubWrite(tid=76, reqid=client.4509.0:236, at_version=0'0, trim_to=0'0, roll_forward_to=32'29))", 2024-04-19T02:03:25.560 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.668971+0000", 2024-04-19T02:03:25.560 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 8.8785532010000008, 2024-04-19T02:03:25.561 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0032027430000000001, 2024-04-19T02:03:25.561 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.561 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.561 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.561 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.561 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.561 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.561 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.668971+0000", 2024-04-19T02:03:25.561 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.561 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.561 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.561 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.561 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.668971+0000", 2024-04-19T02:03:25.561 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.562 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.562 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.562 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.562 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.668981+0000", 2024-04-19T02:03:25.562 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.5689999999999994e-06 2024-04-19T02:03:25.562 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.562 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.562 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.562 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.669004+0000", 2024-04-19T02:03:25.562 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2637e-05 2024-04-19T02:03:25.562 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.562 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.562 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.562 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.669054+0000", 2024-04-19T02:03:25.562 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.0195000000000003e-05 2024-04-19T02:03:25.563 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.563 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.563 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.563 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.669071+0000", 2024-04-19T02:03:25.563 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6812000000000001e-05 2024-04-19T02:03:25.563 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.563 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.563 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.563 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.669298+0000", 2024-04-19T02:03:25.563 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00022734 2024-04-19T02:03:25.563 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.563 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.563 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.563 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.669313+0000", 2024-04-19T02:03:25.564 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4986e-05 2024-04-19T02:03:25.564 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.564 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.564 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.564 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.672120+0000", 2024-04-19T02:03:25.564 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0028068289999999998 2024-04-19T02:03:25.564 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.564 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.564 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.564 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.672174+0000", 2024-04-19T02:03:25.564 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.4375e-05 2024-04-19T02:03:25.564 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.564 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.564 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.564 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.565 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.565 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.2s5 32/26 ECSubWrite(tid=77, reqid=osd.3.0:77, at_version=32'31, trim_to=0'0, roll_forward_to=32'29))", 2024-04-19T02:03:25.565 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.672052+0000", 2024-04-19T02:03:25.565 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 8.8754722560000001, 2024-04-19T02:03:25.565 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0025508480000000001, 2024-04-19T02:03:25.565 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.565 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.565 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.565 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.565 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.565 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.565 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.672052+0000", 2024-04-19T02:03:25.565 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.565 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.565 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.565 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.566 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.672052+0000", 2024-04-19T02:03:25.566 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.566 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.566 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.566 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.566 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.672055+0000", 2024-04-19T02:03:25.566 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3740000000000001e-06 2024-04-19T02:03:25.566 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.566 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.566 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.566 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.672208+0000", 2024-04-19T02:03:25.566 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000153361 2024-04-19T02:03:25.566 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.566 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.567 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.567 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.672250+0000", 2024-04-19T02:03:25.567 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2182999999999999e-05 2024-04-19T02:03:25.567 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.567 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.567 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.567 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.672276+0000", 2024-04-19T02:03:25.567 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5882000000000001e-05 2024-04-19T02:03:25.567 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.567 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.567 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.567 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.672452+0000", 2024-04-19T02:03:25.567 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00017611800000000001 2024-04-19T02:03:25.567 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.568 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.568 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.568 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.672471+0000", 2024-04-19T02:03:25.568 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8791999999999999e-05 2024-04-19T02:03:25.568 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.568 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.568 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.568 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.674508+0000", 2024-04-19T02:03:25.568 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0020369910000000001 2024-04-19T02:03:25.568 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.568 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.568 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.568 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.674603+0000", 2024-04-19T02:03:25.568 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.5147000000000002e-05 2024-04-19T02:03:25.568 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.569 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.569 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.569 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.569 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.569 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.2s5 32/26 ECSubWrite(tid=78, reqid=osd.3.0:78, at_version=32'33, trim_to=0'0, roll_forward_to=32'29))", 2024-04-19T02:03:25.569 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.673763+0000", 2024-04-19T02:03:25.569 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 8.8737616989999992, 2024-04-19T02:03:25.569 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0031152150000000002, 2024-04-19T02:03:25.569 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.569 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.569 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.569 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.569 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.569 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.569 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.673763+0000", 2024-04-19T02:03:25.569 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.570 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.570 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.570 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.570 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.673763+0000", 2024-04-19T02:03:25.570 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.570 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.570 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.570 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.570 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.673764+0000", 2024-04-19T02:03:25.570 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.547e-06 2024-04-19T02:03:25.570 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.570 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.570 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.570 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.673833+0000", 2024-04-19T02:03:25.571 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8281000000000006e-05 2024-04-19T02:03:25.571 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.571 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.571 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.571 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.673867+0000", 2024-04-19T02:03:25.571 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3918999999999998e-05 2024-04-19T02:03:25.571 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.571 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.571 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.571 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.673888+0000", 2024-04-19T02:03:25.571 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1713e-05 2024-04-19T02:03:25.571 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.571 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.571 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.571 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.674038+0000", 2024-04-19T02:03:25.572 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000149851 2024-04-19T02:03:25.572 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.572 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.572 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.572 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.674052+0000", 2024-04-19T02:03:25.572 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3702e-05 2024-04-19T02:03:25.572 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.572 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.572 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.572 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.676798+0000", 2024-04-19T02:03:25.572 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0027456009999999999 2024-04-19T02:03:25.572 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.572 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.572 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.572 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.676878+0000", 2024-04-19T02:03:25.572 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0600999999999996e-05 2024-04-19T02:03:25.573 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.573 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.573 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.573 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.573 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.573 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.2s5 32/26 ECSubWrite(tid=79, reqid=osd.3.0:78, at_version=0'0, trim_to=0'0, roll_forward_to=32'33))", 2024-04-19T02:03:25.573 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.680696+0000", 2024-04-19T02:03:25.573 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 8.866828752, 2024-04-19T02:03:25.573 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0034300160000000001, 2024-04-19T02:03:25.573 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.573 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.573 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.573 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.573 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.573 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.574 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.680696+0000", 2024-04-19T02:03:25.574 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.574 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.574 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.574 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.574 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.680696+0000", 2024-04-19T02:03:25.574 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.574 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.574 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.574 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.574 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.680697+0000", 2024-04-19T02:03:25.574 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5659999999999999e-06 2024-04-19T02:03:25.574 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.574 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.575 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.575 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.680717+0000", 2024-04-19T02:03:25.575 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9122000000000001e-05 2024-04-19T02:03:25.575 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.575 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.575 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.575 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.680737+0000", 2024-04-19T02:03:25.575 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0429999999999999e-05 2024-04-19T02:03:25.575 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.575 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.575 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.575 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.680750+0000", 2024-04-19T02:03:25.575 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3499000000000001e-05 2024-04-19T02:03:25.575 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.575 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.576 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.576 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.680852+0000", 2024-04-19T02:03:25.576 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000101403 2024-04-19T02:03:25.576 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.576 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.576 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.576 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.680869+0000", 2024-04-19T02:03:25.576 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6937999999999999e-05 2024-04-19T02:03:25.576 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.576 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.576 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.576 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.684084+0000", 2024-04-19T02:03:25.576 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0032153009999999998 2024-04-19T02:03:25.576 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.577 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.577 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.577 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.684126+0000", 2024-04-19T02:03:25.577 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1757e-05 2024-04-19T02:03:25.577 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.577 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.577 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.577 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.577 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.577 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.2s5 35/26 ECSubWrite(tid=92, reqid=osd.3.0:92, at_version=35'46, trim_to=0'0, roll_forward_to=34'44))", 2024-04-19T02:03:25.577 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.663020+0000", 2024-04-19T02:03:25.577 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 5.8845048110000002, 2024-04-19T02:03:25.577 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00323012, 2024-04-19T02:03:25.577 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.577 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.578 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.578 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.578 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.578 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.578 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.663020+0000", 2024-04-19T02:03:25.578 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.578 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.578 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.578 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.578 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.663020+0000", 2024-04-19T02:03:25.578 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.578 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.578 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.578 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.578 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.663020+0000", 2024-04-19T02:03:25.579 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.03e-07 2024-04-19T02:03:25.579 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.579 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.579 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.579 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.663045+0000", 2024-04-19T02:03:25.579 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5017000000000001e-05 2024-04-19T02:03:25.579 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.579 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.579 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.579 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.663060+0000", 2024-04-19T02:03:25.579 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4707e-05 2024-04-19T02:03:25.579 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.579 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.579 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.579 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.663769+0000", 2024-04-19T02:03:25.580 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00070962800000000002 2024-04-19T02:03:25.580 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.580 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.580 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.580 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.663845+0000", 2024-04-19T02:03:25.580 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.5989999999999996e-05 2024-04-19T02:03:25.580 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.580 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.580 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.580 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.663859+0000", 2024-04-19T02:03:25.580 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3567000000000001e-05 2024-04-19T02:03:25.580 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.580 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.580 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.580 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.666223+0000", 2024-04-19T02:03:25.581 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0023643380000000001 2024-04-19T02:03:25.581 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.581 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.581 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.581 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.666250+0000", 2024-04-19T02:03:25.581 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6570000000000001e-05 2024-04-19T02:03:25.581 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.581 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.581 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.581 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.581 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.581 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.7s7 35/26 ECSubWrite(tid=93, reqid=osd.3.0:93, at_version=35'6, trim_to=0'0, roll_forward_to=34'4))", 2024-04-19T02:03:25.581 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.663812+0000", 2024-04-19T02:03:25.581 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 5.8837127269999998, 2024-04-19T02:03:25.581 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0024422329999999998, 2024-04-19T02:03:25.582 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.582 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.582 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.582 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.582 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.582 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.582 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.663812+0000", 2024-04-19T02:03:25.582 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.582 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.582 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.582 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.582 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.663812+0000", 2024-04-19T02:03:25.582 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.582 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.582 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.583 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.583 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.663812+0000", 2024-04-19T02:03:25.583 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.3700000000000001e-07 2024-04-19T02:03:25.583 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.583 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.583 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.583 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.663833+0000", 2024-04-19T02:03:25.583 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0298000000000001e-05 2024-04-19T02:03:25.583 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.583 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.583 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.583 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.663845+0000", 2024-04-19T02:03:25.583 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1972e-05 2024-04-19T02:03:25.583 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.583 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.584 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.584 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.663852+0000", 2024-04-19T02:03:25.584 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.8469999999999996e-06 2024-04-19T02:03:25.584 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.584 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.584 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.584 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.663915+0000", 2024-04-19T02:03:25.584 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.2255999999999995e-05 2024-04-19T02:03:25.584 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.584 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.584 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.584 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.663924+0000", 2024-04-19T02:03:25.584 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.7310000000000005e-06 2024-04-19T02:03:25.584 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.585 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.585 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.585 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.666231+0000", 2024-04-19T02:03:25.585 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0023065490000000002 2024-04-19T02:03:25.585 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.585 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.585 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.585 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.666254+0000", 2024-04-19T02:03:25.585 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3142999999999998e-05 2024-04-19T02:03:25.585 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.585 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.585 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.585 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.585 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.585 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.8s1 35/26 ECSubWrite(tid=60, reqid=osd.4.0:60, at_version=35'24, trim_to=0'0, roll_forward_to=34'22))", 2024-04-19T02:03:25.585 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.689661+0000", 2024-04-19T02:03:25.586 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 5.8578631630000002, 2024-04-19T02:03:25.586 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0036629990000000001, 2024-04-19T02:03:25.586 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.586 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.586 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.586 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.586 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.586 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.586 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.689661+0000", 2024-04-19T02:03:25.586 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.586 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.586 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.586 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.586 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.689661+0000", 2024-04-19T02:03:25.587 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.587 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.587 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.587 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.587 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.689662+0000", 2024-04-19T02:03:25.587 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7300000000000002e-07 2024-04-19T02:03:25.587 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.587 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.587 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.587 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.689684+0000", 2024-04-19T02:03:25.588 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2166000000000001e-05 2024-04-19T02:03:25.588 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.588 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.588 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.588 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.689702+0000", 2024-04-19T02:03:25.588 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7728999999999999e-05 2024-04-19T02:03:25.588 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.588 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.588 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.588 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.689711+0000", 2024-04-19T02:03:25.588 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.4560000000000006e-06 2024-04-19T02:03:25.588 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.588 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.588 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.588 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.689789+0000", 2024-04-19T02:03:25.589 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.8242000000000001e-05 2024-04-19T02:03:25.589 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.589 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.589 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.589 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.689799+0000", 2024-04-19T02:03:25.589 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3149999999999998e-06 2024-04-19T02:03:25.589 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.589 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.589 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.589 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.693289+0000", 2024-04-19T02:03:25.589 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0034902430000000001 2024-04-19T02:03:25.589 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.589 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.590 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.590 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.693324+0000", 2024-04-19T02:03:25.590 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5574999999999997e-05 2024-04-19T02:03:25.590 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.590 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.590 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.590 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.590 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.590 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.as2 35/26 ECSubWrite(tid=59, reqid=client.4509.0:285, at_version=35'23, trim_to=0'0, roll_forward_to=34'22))", 2024-04-19T02:03:25.590 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.754882+0000", 2024-04-19T02:03:25.590 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 5.7926423160000002, 2024-04-19T02:03:25.590 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002948619, 2024-04-19T02:03:25.590 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.590 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.590 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.591 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.591 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.591 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.591 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.754882+0000", 2024-04-19T02:03:25.591 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.591 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.591 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.591 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.591 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.754882+0000", 2024-04-19T02:03:25.591 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.591 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.591 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.591 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.591 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.754883+0000", 2024-04-19T02:03:25.591 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.8400000000000003e-07 2024-04-19T02:03:25.592 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.592 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.592 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.592 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.754978+0000", 2024-04-19T02:03:25.592 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.4450999999999994e-05 2024-04-19T02:03:25.592 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.592 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.592 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.592 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.754990+0000", 2024-04-19T02:03:25.592 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2172e-05 2024-04-19T02:03:25.592 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.592 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.592 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.592 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.754999+0000", 2024-04-19T02:03:25.593 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0999999999999993e-06 2024-04-19T02:03:25.593 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.593 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.593 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.593 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.755060+0000", 2024-04-19T02:03:25.593 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.1216999999999993e-05 2024-04-19T02:03:25.593 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.593 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.593 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.593 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.755070+0000", 2024-04-19T02:03:25.593 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9040000000000004e-06 2024-04-19T02:03:25.593 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.593 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.593 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.593 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.757792+0000", 2024-04-19T02:03:25.594 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0027216860000000001 2024-04-19T02:03:25.594 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.594 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.594 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.594 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.757831+0000", 2024-04-19T02:03:25.594 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9304999999999997e-05 2024-04-19T02:03:25.594 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.594 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.594 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.594 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.594 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.594 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.5s5 37/26 ECSubWrite(tid=38, reqid=client.4509.0:293, at_version=37'17, trim_to=0'0, roll_forward_to=35'15))", 2024-04-19T02:03:25.594 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:21.694676+0000", 2024-04-19T02:03:25.594 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 3.8528483539999998, 2024-04-19T02:03:25.595 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0033774370000000001, 2024-04-19T02:03:25.595 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.595 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.595 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.595 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.595 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.595 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.595 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.694676+0000", 2024-04-19T02:03:25.595 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.595 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.595 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.595 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.595 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.694676+0000", 2024-04-19T02:03:25.595 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.595 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.595 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.596 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.596 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.694677+0000", 2024-04-19T02:03:25.596 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2300000000000002e-07 2024-04-19T02:03:25.596 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.596 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.596 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.596 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.694720+0000", 2024-04-19T02:03:25.596 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.3263999999999998e-05 2024-04-19T02:03:25.596 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.596 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.596 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.596 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.694765+0000", 2024-04-19T02:03:25.596 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4731000000000003e-05 2024-04-19T02:03:25.596 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.596 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.597 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.597 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.694788+0000", 2024-04-19T02:03:25.597 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3777999999999998e-05 2024-04-19T02:03:25.597 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.597 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.597 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.597 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.695052+0000", 2024-04-19T02:03:25.597 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00026359000000000002 2024-04-19T02:03:25.597 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.597 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.597 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.597 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.695079+0000", 2024-04-19T02:03:25.597 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7365e-05 2024-04-19T02:03:25.597 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.597 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.597 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.598 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.697964+0000", 2024-04-19T02:03:25.598 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0028846280000000002 2024-04-19T02:03:25.598 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.598 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.598 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.598 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.698054+0000", 2024-04-19T02:03:25.598 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.9758000000000004e-05 2024-04-19T02:03:25.598 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.598 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.598 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.598 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.598 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.598 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.5s5 37/26 ECSubWrite(tid=39, reqid=client.4509.0:293, at_version=0'0, trim_to=0'0, roll_forward_to=37'17))", 2024-04-19T02:03:25.598 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:21.700630+0000", 2024-04-19T02:03:25.598 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 3.8468943109999998, 2024-04-19T02:03:25.599 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002064382, 2024-04-19T02:03:25.599 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.599 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.599 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.599 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.599 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.599 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.599 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.700630+0000", 2024-04-19T02:03:25.599 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.599 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.599 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.599 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.599 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.700630+0000", 2024-04-19T02:03:25.599 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.599 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.600 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.600 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.600 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.700632+0000", 2024-04-19T02:03:25.600 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9460000000000001e-06 2024-04-19T02:03:25.600 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.600 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.600 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.600 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.700657+0000", 2024-04-19T02:03:25.600 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4692999999999999e-05 2024-04-19T02:03:25.600 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.600 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.600 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.600 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.700687+0000", 2024-04-19T02:03:25.600 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0238000000000001e-05 2024-04-19T02:03:25.600 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.601 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.601 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.601 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.700721+0000", 2024-04-19T02:03:25.601 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3614e-05 2024-04-19T02:03:25.601 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.601 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.601 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.601 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.700883+0000", 2024-04-19T02:03:25.601 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00016268599999999999 2024-04-19T02:03:25.601 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.601 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.601 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.601 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.700906+0000", 2024-04-19T02:03:25.601 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2086000000000002e-05 2024-04-19T02:03:25.601 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.602 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.602 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.602 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.702564+0000", 2024-04-19T02:03:25.602 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00165858 2024-04-19T02:03:25.602 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.602 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.602 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.602 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.702695+0000", 2024-04-19T02:03:25.602 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013053899999999999 2024-04-19T02:03:25.602 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.602 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.602 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.602 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.602 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.603 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.8s1 37/26 ECSubWrite(tid=63, reqid=client.4509.0:294, at_version=37'28, trim_to=0'0, roll_forward_to=35'26))", 2024-04-19T02:03:25.603 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:21.715952+0000", 2024-04-19T02:03:25.603 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 3.8315729469999997, 2024-04-19T02:03:25.603 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0031547630000000001, 2024-04-19T02:03:25.603 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.603 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.603 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.603 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.603 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.603 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.603 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.715952+0000", 2024-04-19T02:03:25.603 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.603 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.603 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.603 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.603 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.715952+0000", 2024-04-19T02:03:25.604 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.604 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.604 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.604 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.604 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.715952+0000", 2024-04-19T02:03:25.604 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.0299999999999999e-07 2024-04-19T02:03:25.604 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.604 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.604 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.604 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.716219+0000", 2024-04-19T02:03:25.604 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00026670399999999999 2024-04-19T02:03:25.604 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.604 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.604 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.604 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.716245+0000", 2024-04-19T02:03:25.605 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6461e-05 2024-04-19T02:03:25.605 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.605 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.605 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.605 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.716260+0000", 2024-04-19T02:03:25.605 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5159e-05 2024-04-19T02:03:25.605 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.605 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.605 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.605 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.716429+0000", 2024-04-19T02:03:25.605 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00016888600000000001 2024-04-19T02:03:25.605 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.606 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.606 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.606 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.716461+0000", 2024-04-19T02:03:25.606 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1758000000000003e-05 2024-04-19T02:03:25.606 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.606 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.606 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.606 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.719017+0000", 2024-04-19T02:03:25.606 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0025554869999999999 2024-04-19T02:03:25.606 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.606 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.606 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.606 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.719106+0000", 2024-04-19T02:03:25.606 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.9804999999999998e-05 2024-04-19T02:03:25.606 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.607 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.607 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.607 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.607 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.607 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:344 3.4s0 3:2277bf56:::smithi05930959-44:head [append 0~606208] snapc 8=[8,6,5] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:25.607 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.735396+0000", 2024-04-19T02:03:25.607 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.81212892800000003, 2024-04-19T02:03:25.607 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.014124197999999999, 2024-04-19T02:03:25.607 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.607 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.607 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:25.607 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:25.607 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:25.607 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:25.608 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 344 2024-04-19T02:03:25.608 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.608 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.608 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.608 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.608 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.735396+0000", 2024-04-19T02:03:25.608 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.608 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.608 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.608 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.608 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.735396+0000", 2024-04-19T02:03:25.608 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.608 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.608 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.608 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.608 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.735399+0000", 2024-04-19T02:03:25.609 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.32e-06 2024-04-19T02:03:25.609 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.609 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.609 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.609 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.736033+0000", 2024-04-19T02:03:25.609 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000633689 2024-04-19T02:03:25.609 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.609 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.609 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.609 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.736074+0000", 2024-04-19T02:03:25.609 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1020999999999999e-05 2024-04-19T02:03:25.609 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.609 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.609 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.609 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.736094+0000", 2024-04-19T02:03:25.610 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0339e-05 2024-04-19T02:03:25.610 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.610 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.610 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.610 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.736202+0000", 2024-04-19T02:03:25.610 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000107899 2024-04-19T02:03:25.610 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.610 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.610 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "op must wait for map", 2024-04-19T02:03:25.610 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.736212+0000", 2024-04-19T02:03:25.610 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.7259999999999997e-06 2024-04-19T02:03:25.610 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.610 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.610 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.610 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.738290+0000", 2024-04-19T02:03:25.611 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002078725 2024-04-19T02:03:25.611 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.611 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.611 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:25.611 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.738359+0000", 2024-04-19T02:03:25.611 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8177999999999997e-05 2024-04-19T02:03:25.611 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.611 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.611 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.611 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.739412+0000", 2024-04-19T02:03:25.611 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0010534240000000001 2024-04-19T02:03:25.611 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.611 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.611 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.611 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.741318+0000", 2024-04-19T02:03:25.612 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001906151 2024-04-19T02:03:25.612 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.612 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.612 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:25.612 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749459+0000", 2024-04-19T02:03:25.612 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0081410830000000003 2024-04-19T02:03:25.612 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.612 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.612 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.612 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749520+0000", 2024-04-19T02:03:25.612 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0643000000000002e-05 2024-04-19T02:03:25.612 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.612 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.612 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.612 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.613 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.613 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:345 3.4s0 3:2277bf56:::smithi05930959-44:head [append 0~720896] snapc 8=[8,6,5] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:25.613 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.745708+0000", 2024-04-19T02:03:25.613 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.80181637400000005, 2024-04-19T02:03:25.613 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0064257389999999998, 2024-04-19T02:03:25.613 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.613 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.613 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:25.613 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:25.613 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:25.613 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:25.613 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 345 2024-04-19T02:03:25.613 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.613 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.613 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.613 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.614 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.745708+0000", 2024-04-19T02:03:25.614 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.614 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.614 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.614 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.614 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.745708+0000", 2024-04-19T02:03:25.614 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.614 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.614 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.614 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.614 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.745710+0000", 2024-04-19T02:03:25.614 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6619999999999999e-06 2024-04-19T02:03:25.614 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.614 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.615 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.615 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746146+0000", 2024-04-19T02:03:25.615 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00043585500000000002 2024-04-19T02:03:25.615 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.615 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.615 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.615 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746160+0000", 2024-04-19T02:03:25.615 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4460999999999999e-05 2024-04-19T02:03:25.615 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.615 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.615 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.615 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746169+0000", 2024-04-19T02:03:25.615 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.9369999999999993e-06 2024-04-19T02:03:25.615 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.616 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.616 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.616 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746255+0000", 2024-04-19T02:03:25.616 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5756999999999999e-05 2024-04-19T02:03:25.616 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.616 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.616 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:25.616 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746358+0000", 2024-04-19T02:03:25.616 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000102722 2024-04-19T02:03:25.616 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.616 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.616 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.616 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.747578+0000", 2024-04-19T02:03:25.616 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001220643 2024-04-19T02:03:25.616 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.617 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.617 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.617 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749531+0000", 2024-04-19T02:03:25.617 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001952454 2024-04-19T02:03:25.617 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.617 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.617 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:25.617 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.752059+0000", 2024-04-19T02:03:25.617 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002528218 2024-04-19T02:03:25.617 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.617 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.617 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.617 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.752134+0000", 2024-04-19T02:03:25.617 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.5030000000000005e-05 2024-04-19T02:03:25.617 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.618 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.618 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.618 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.618 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.618 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:346 3.4s0 3:2277bf56:::smithi05930959-44:head [append 0~49152] snapc 8=[8,6,5] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:25.618 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.746194+0000", 2024-04-19T02:03:25.618 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.80133069300000004, 2024-04-19T02:03:25.618 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.009380457, 2024-04-19T02:03:25.618 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.618 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.618 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:25.618 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:25.618 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:25.618 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:25.619 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 346 2024-04-19T02:03:25.619 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.619 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.619 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.619 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.619 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746194+0000", 2024-04-19T02:03:25.619 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.619 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.619 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.619 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.619 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746194+0000", 2024-04-19T02:03:25.619 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.619 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.619 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.619 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.620 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746195+0000", 2024-04-19T02:03:25.620 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.6199999999999997e-07 2024-04-19T02:03:25.620 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.620 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.620 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.620 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746238+0000", 2024-04-19T02:03:25.620 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.3105999999999997e-05 2024-04-19T02:03:25.620 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.620 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.620 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.620 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746247+0000", 2024-04-19T02:03:25.620 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.1540000000000008e-06 2024-04-19T02:03:25.620 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.620 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.620 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.620 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746257+0000", 2024-04-19T02:03:25.621 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0558e-05 2024-04-19T02:03:25.621 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.621 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.621 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.621 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.748250+0000", 2024-04-19T02:03:25.621 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0019923380000000002 2024-04-19T02:03:25.621 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.621 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.621 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:25.621 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.748349+0000", 2024-04-19T02:03:25.621 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9030999999999994e-05 2024-04-19T02:03:25.621 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.621 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.621 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.621 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.748987+0000", 2024-04-19T02:03:25.622 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00063871800000000003 2024-04-19T02:03:25.622 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.622 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.622 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.622 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750226+0000", 2024-04-19T02:03:25.622 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0012389250000000001 2024-04-19T02:03:25.622 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.622 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.622 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:25.622 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.755523+0000", 2024-04-19T02:03:25.622 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.005297024 2024-04-19T02:03:25.622 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.622 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.622 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.622 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.755574+0000", 2024-04-19T02:03:25.623 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.0940999999999999e-05 2024-04-19T02:03:25.623 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.623 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.623 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.623 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.623 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.623 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:347 3.4s0 3:2277bf56:::smithi05930959-44:head [setxattr _header (56) in=7b] snapc 8=[8,6,5] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:25.623 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.746274+0000", 2024-04-19T02:03:25.623 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.80125051300000005, 2024-04-19T02:03:25.623 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010236558, 2024-04-19T02:03:25.623 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.623 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.623 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:25.623 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:25.623 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:25.624 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:25.624 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 347 2024-04-19T02:03:25.624 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.624 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.624 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.624 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.624 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746274+0000", 2024-04-19T02:03:25.624 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.624 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.624 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.624 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.624 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746274+0000", 2024-04-19T02:03:25.624 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.624 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.624 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.625 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.625 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746275+0000", 2024-04-19T02:03:25.625 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.3900000000000004e-07 2024-04-19T02:03:25.625 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.625 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.625 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.625 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746279+0000", 2024-04-19T02:03:25.625 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0679999999999998e-06 2024-04-19T02:03:25.625 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.625 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.625 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.625 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746299+0000", 2024-04-19T02:03:25.625 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9641e-05 2024-04-19T02:03:25.625 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.625 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.625 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.626 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746307+0000", 2024-04-19T02:03:25.626 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2570000000000007e-06 2024-04-19T02:03:25.626 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.626 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.626 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.626 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749555+0000", 2024-04-19T02:03:25.626 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0032478519999999999 2024-04-19T02:03:25.626 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.626 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.626 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:25.626 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749610+0000", 2024-04-19T02:03:25.626 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5155999999999998e-05 2024-04-19T02:03:25.626 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.626 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.627 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:25.627 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749969+0000", 2024-04-19T02:03:25.627 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00035909400000000001 2024-04-19T02:03:25.627 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.627 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.627 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:25.627 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750800+0000", 2024-04-19T02:03:25.627 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00083078000000000004 2024-04-19T02:03:25.627 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.627 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.627 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:25.627 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.756433+0000", 2024-04-19T02:03:25.627 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0056334009999999997 2024-04-19T02:03:25.627 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.627 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.628 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.628 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.756511+0000", 2024-04-19T02:03:25.628 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.7570000000000004e-05 2024-04-19T02:03:25.628 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.628 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.628 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.628 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.628 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.628 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:348 3.4s0 3:2277bf56:::smithi05930959-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:25.628 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.746345+0000", 2024-04-19T02:03:25.628 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.80117970800000005, 2024-04-19T02:03:25.628 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012391231000000001, 2024-04-19T02:03:25.628 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.628 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.628 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:25.629 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:25.629 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:25.629 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:25.629 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 348 2024-04-19T02:03:25.629 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.629 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.629 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.629 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.629 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746345+0000", 2024-04-19T02:03:25.629 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.629 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.629 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.629 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.629 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746345+0000", 2024-04-19T02:03:25.629 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.630 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.630 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.630 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.630 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746346+0000", 2024-04-19T02:03:25.630 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.5899999999999996e-07 2024-04-19T02:03:25.630 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.630 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.630 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.630 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746349+0000", 2024-04-19T02:03:25.630 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2100000000000002e-06 2024-04-19T02:03:25.630 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.630 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.630 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.630 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746359+0000", 2024-04-19T02:03:25.630 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.8069999999999994e-06 2024-04-19T02:03:25.631 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.631 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.631 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.631 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746366+0000", 2024-04-19T02:03:25.631 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.1969999999999999e-06 2024-04-19T02:03:25.631 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.631 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.631 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.631 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750173+0000", 2024-04-19T02:03:25.631 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0038071400000000001 2024-04-19T02:03:25.631 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.631 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.631 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:25.631 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750223+0000", 2024-04-19T02:03:25.631 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9597e-05 2024-04-19T02:03:25.632 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.632 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.632 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.632 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.757010+0000", 2024-04-19T02:03:25.632 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.006787062 2024-04-19T02:03:25.632 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.632 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.632 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:25.632 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.757155+0000", 2024-04-19T02:03:25.632 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000145303 2024-04-19T02:03:25.632 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.632 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.632 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.632 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.758736+0000", 2024-04-19T02:03:25.632 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0015812560000000001 2024-04-19T02:03:25.632 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.633 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.633 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.633 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.633 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.633 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.4s0 40/26 ECSubWriteReply(tid=37, last_complete=40'25, committed=1, applied=1))", 2024-04-19T02:03:25.633 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.746396+0000", 2024-04-19T02:03:25.633 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.80112828199999997, 2024-04-19T02:03:25.633 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0031359339999999999, 2024-04-19T02:03:25.633 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:25.633 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:25.633 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:25.633 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:25.633 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.633 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:25.633 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746396+0000", 2024-04-19T02:03:25.634 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.634 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.634 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.634 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:25.634 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746396+0000", 2024-04-19T02:03:25.634 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:25.634 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.634 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.634 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:25.634 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746397+0000", 2024-04-19T02:03:25.634 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.6400000000000002e-07 2024-04-19T02:03:25.634 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.634 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.634 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:25.635 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746400+0000", 2024-04-19T02:03:25.635 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5149999999999998e-06 2024-04-19T02:03:25.635 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.635 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.635 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:25.635 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746409+0000", 2024-04-19T02:03:25.635 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2260000000000005e-06 2024-04-19T02:03:25.635 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.635 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.635 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:25.635 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746415+0000", 2024-04-19T02:03:25.635 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.7870000000000004e-06 2024-04-19T02:03:25.635 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.635 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.635 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:25.636 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749381+0000", 2024-04-19T02:03:25.636 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002965975 2024-04-19T02:03:25.636 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:25.636 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:25.636 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:25.636 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749532+0000", 2024-04-19T02:03:25.636 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000150867 2024-04-19T02:03:25.636 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.636 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.636 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.636 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:25.636 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:25.636 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:25.637 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:25.547+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:25.638 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-19T02:03:25.648 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:25.650+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:25.705 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-19T02:03:25.706 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-19T02:03:25.730 INFO:tasks.rados.rados.0.smithi059.stdout:134: done (4 left) 2024-04-19T02:03:25.730 INFO:tasks.rados.rados.0.smithi059.stdout:135: done (3 left) 2024-04-19T02:03:25.730 INFO:tasks.rados.rados.0.smithi059.stdout:136: done (2 left) 2024-04-19T02:03:25.730 INFO:tasks.rados.rados.0.smithi059.stdout:137: done (1 left) 2024-04-19T02:03:25.730 INFO:tasks.rados.rados.0.smithi059.stdout:138: done (0 left) 2024-04-19T02:03:25.730 INFO:tasks.rados.rados.0.smithi059.stdout:139: snap_remove snap 3 2024-04-19T02:03:25.742 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-19T02:03:25.749 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:25.751+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:25.818 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:25.818 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:03:25.818 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-04-19T02:03:25.818 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:25.835 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-19T02:03:25.882 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:25.883+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:25.951 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:25.951+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:26.002 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:26.002 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:03:26.003 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-04-19T02:03:26.003 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-04-19T02:03:26.003 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:26.017 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-19T02:03:26.052 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:26.055+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:26.151 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:26.151 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-04-19T02:03:26.151 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-04-19T02:03:26.152 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-04-19T02:03:26.152 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.152 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "pg_update_log_missing(3.es5 epoch 32/26 rep_tid 18 entries 32'12 (0'0) error 3:79fe7502:::smithi05930959-12:head by client.4509.0:240 0.000000 -2 [r=-2+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 30'10)", 2024-04-19T02:03:26.152 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.651404+0000", 2024-04-19T02:03:26.152 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 9.502651041, 2024-04-19T02:03:26.152 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001738641, 2024-04-19T02:03:26.152 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.152 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.152 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:26.152 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.152 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.152 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.152 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.651404+0000", 2024-04-19T02:03:26.152 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.152 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.153 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.153 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.153 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.651404+0000", 2024-04-19T02:03:26.153 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.153 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.153 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.153 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.153 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.651405+0000", 2024-04-19T02:03:26.153 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.5000000000000002e-07 2024-04-19T02:03:26.153 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.153 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.153 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.153 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.651428+0000", 2024-04-19T02:03:26.153 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3523000000000002e-05 2024-04-19T02:03:26.153 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.154 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.154 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.154 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.651496+0000", 2024-04-19T02:03:26.154 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8108000000000006e-05 2024-04-19T02:03:26.154 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.154 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.154 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.154 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.651594+0000", 2024-04-19T02:03:26.154 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.8016999999999993e-05 2024-04-19T02:03:26.154 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.154 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.154 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.154 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.651832+0000", 2024-04-19T02:03:26.154 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00023731300000000001 2024-04-19T02:03:26.155 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.155 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.155 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.155 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.653143+0000", 2024-04-19T02:03:26.155 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0013110299999999999 2024-04-19T02:03:26.155 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.155 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.155 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.155 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.155 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.155 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s5 35/26 ECSubWrite(tid=29, reqid=osd.1.0:29, at_version=35'18, trim_to=0'0, roll_forward_to=34'16))", 2024-04-19T02:03:26.155 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.667338+0000", 2024-04-19T02:03:26.155 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 6.4867168790000003, 2024-04-19T02:03:26.155 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002196726, 2024-04-19T02:03:26.155 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.156 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.156 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:26.156 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.156 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.156 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.156 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.667338+0000", 2024-04-19T02:03:26.156 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.156 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.156 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.156 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.156 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.667338+0000", 2024-04-19T02:03:26.156 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.156 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.156 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.156 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.157 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.667338+0000", 2024-04-19T02:03:26.157 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2700000000000001e-07 2024-04-19T02:03:26.157 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.157 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.157 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.157 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.667362+0000", 2024-04-19T02:03:26.157 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3343e-05 2024-04-19T02:03:26.157 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.157 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.157 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.157 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.667375+0000", 2024-04-19T02:03:26.157 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3750999999999999e-05 2024-04-19T02:03:26.157 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.157 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.157 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.158 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.667386+0000", 2024-04-19T02:03:26.158 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0159e-05 2024-04-19T02:03:26.158 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.158 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.158 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.158 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.667497+0000", 2024-04-19T02:03:26.158 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000111229 2024-04-19T02:03:26.158 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.158 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.158 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:26.158 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.667507+0000", 2024-04-19T02:03:26.158 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0387000000000001e-05 2024-04-19T02:03:26.158 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.158 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.159 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:26.159 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.669488+0000", 2024-04-19T02:03:26.159 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0019807650000000002 2024-04-19T02:03:26.159 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.159 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.159 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.159 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.669535+0000", 2024-04-19T02:03:26.159 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.6864999999999999e-05 2024-04-19T02:03:26.159 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.159 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.159 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.159 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.159 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.159 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es5 35/26 ECSubWrite(tid=23, reqid=osd.7.0:23, at_version=35'18, trim_to=0'0, roll_forward_to=34'16))", 2024-04-19T02:03:26.159 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.674564+0000", 2024-04-19T02:03:26.160 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 6.479491039, 2024-04-19T02:03:26.160 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0023738539999999999, 2024-04-19T02:03:26.160 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.160 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.160 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:26.160 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.160 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.160 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.160 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.674564+0000", 2024-04-19T02:03:26.160 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.160 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.160 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.160 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.160 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.674564+0000", 2024-04-19T02:03:26.160 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.161 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.161 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.161 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.161 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.674564+0000", 2024-04-19T02:03:26.161 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.3599999999999999e-07 2024-04-19T02:03:26.161 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.161 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.161 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.161 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.674595+0000", 2024-04-19T02:03:26.161 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0273999999999998e-05 2024-04-19T02:03:26.161 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.161 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.161 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.161 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.674614+0000", 2024-04-19T02:03:26.162 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9147000000000002e-05 2024-04-19T02:03:26.162 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.162 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.162 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.162 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.674625+0000", 2024-04-19T02:03:26.162 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1056999999999999e-05 2024-04-19T02:03:26.162 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.162 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.162 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.162 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.674723+0000", 2024-04-19T02:03:26.162 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.8010999999999994e-05 2024-04-19T02:03:26.162 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.162 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.162 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:26.162 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.674733+0000", 2024-04-19T02:03:26.163 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.045e-05 2024-04-19T02:03:26.163 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.163 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.163 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:26.163 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.676896+0000", 2024-04-19T02:03:26.163 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002162481 2024-04-19T02:03:26.163 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.163 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.163 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.163 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.676938+0000", 2024-04-19T02:03:26.163 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1998e-05 2024-04-19T02:03:26.163 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.163 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.163 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.163 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.164 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.164 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s5 35/26 ECSubWrite(tid=30, reqid=osd.1.0:29, at_version=0'0, trim_to=0'0, roll_forward_to=35'18))", 2024-04-19T02:03:26.164 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.680759+0000", 2024-04-19T02:03:26.164 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 6.4732955380000003, 2024-04-19T02:03:26.164 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0036058090000000002, 2024-04-19T02:03:26.164 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.164 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.164 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:26.164 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.164 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.164 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.164 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680759+0000", 2024-04-19T02:03:26.164 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.164 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.165 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.165 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.165 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680759+0000", 2024-04-19T02:03:26.165 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.165 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.165 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.165 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.165 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680761+0000", 2024-04-19T02:03:26.165 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2929999999999999e-06 2024-04-19T02:03:26.165 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.165 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.165 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.165 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680775+0000", 2024-04-19T02:03:26.165 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4284000000000001e-05 2024-04-19T02:03:26.166 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.166 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.166 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.166 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680793+0000", 2024-04-19T02:03:26.166 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8176999999999999e-05 2024-04-19T02:03:26.166 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.166 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.166 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.166 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680804+0000", 2024-04-19T02:03:26.166 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0508e-05 2024-04-19T02:03:26.166 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.166 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.166 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.166 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680903+0000", 2024-04-19T02:03:26.167 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9370999999999994e-05 2024-04-19T02:03:26.167 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.167 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.167 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:26.167 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680914+0000", 2024-04-19T02:03:26.167 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1255e-05 2024-04-19T02:03:26.167 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.167 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.167 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:26.167 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.684325+0000", 2024-04-19T02:03:26.167 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.003410471 2024-04-19T02:03:26.167 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.167 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.167 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.168 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.684365+0000", 2024-04-19T02:03:26.168 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0450000000000001e-05 2024-04-19T02:03:26.168 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.168 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.168 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.168 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.168 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.168 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs2 35/26 ECSubWrite(tid=54, reqid=client.4509.0:271, at_version=35'14, trim_to=0'0, roll_forward_to=34'13))", 2024-04-19T02:03:26.168 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.684827+0000", 2024-04-19T02:03:26.168 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 6.4692277359999997, 2024-04-19T02:03:26.168 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.003378519, 2024-04-19T02:03:26.168 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.168 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.168 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:26.168 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.169 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.169 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.169 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.684827+0000", 2024-04-19T02:03:26.169 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.169 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.169 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.169 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.169 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.684827+0000", 2024-04-19T02:03:26.169 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.169 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.169 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.169 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.169 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.684827+0000", 2024-04-19T02:03:26.169 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7000000000000001e-07 2024-04-19T02:03:26.170 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.170 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.170 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.170 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.685370+0000", 2024-04-19T02:03:26.170 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00054286000000000004 2024-04-19T02:03:26.170 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.170 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.170 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.170 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.685390+0000", 2024-04-19T02:03:26.170 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9346000000000001e-05 2024-04-19T02:03:26.170 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.170 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.170 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.170 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.685400+0000", 2024-04-19T02:03:26.171 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0079e-05 2024-04-19T02:03:26.171 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.171 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.171 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.171 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.685500+0000", 2024-04-19T02:03:26.171 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000100074 2024-04-19T02:03:26.171 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.171 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.171 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:26.171 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.685511+0000", 2024-04-19T02:03:26.171 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1345999999999999e-05 2024-04-19T02:03:26.171 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.171 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.171 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:26.171 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.688159+0000", 2024-04-19T02:03:26.172 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0026475159999999999 2024-04-19T02:03:26.172 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.172 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.172 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.172 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.688206+0000", 2024-04-19T02:03:26.172 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.7027999999999998e-05 2024-04-19T02:03:26.172 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.172 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.172 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.172 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.172 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.173 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs2 35/26 ECSubWrite(tid=55, reqid=client.4509.0:272, at_version=35'15, trim_to=0'0, roll_forward_to=34'13))", 2024-04-19T02:03:26.173 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.689508+0000", 2024-04-19T02:03:26.173 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 6.4645466550000004, 2024-04-19T02:03:26.173 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0029708820000000002, 2024-04-19T02:03:26.173 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.173 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.173 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:26.173 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.173 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.173 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.173 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.689508+0000", 2024-04-19T02:03:26.173 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.173 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.174 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.174 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.174 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.689508+0000", 2024-04-19T02:03:26.174 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.174 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.174 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.174 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.174 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.689510+0000", 2024-04-19T02:03:26.174 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2610000000000001e-06 2024-04-19T02:03:26.174 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.174 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.174 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.174 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.689634+0000", 2024-04-19T02:03:26.175 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000124635 2024-04-19T02:03:26.175 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.175 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.175 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.175 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.689654+0000", 2024-04-19T02:03:26.175 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0024999999999999e-05 2024-04-19T02:03:26.175 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.175 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.175 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.175 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.689664+0000", 2024-04-19T02:03:26.175 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0061000000000001e-05 2024-04-19T02:03:26.175 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.175 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.175 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.175 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.689751+0000", 2024-04-19T02:03:26.175 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.6637000000000004e-05 2024-04-19T02:03:26.176 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.176 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.176 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:26.176 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.689761+0000", 2024-04-19T02:03:26.176 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0261000000000001e-05 2024-04-19T02:03:26.176 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.176 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.176 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:26.176 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.692434+0000", 2024-04-19T02:03:26.176 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0026723490000000001 2024-04-19T02:03:26.177 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.177 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.177 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.177 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.692479+0000", 2024-04-19T02:03:26.177 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5652999999999998e-05 2024-04-19T02:03:26.177 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.177 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.177 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.177 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.177 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.177 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es5 35/26 ECSubWrite(tid=24, reqid=osd.7.0:23, at_version=0'0, trim_to=0'0, roll_forward_to=35'18))", 2024-04-19T02:03:26.177 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.696711+0000", 2024-04-19T02:03:26.177 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 6.4573437739999999, 2024-04-19T02:03:26.177 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002790196, 2024-04-19T02:03:26.178 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.178 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.178 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:26.178 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.178 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.178 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.178 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.696711+0000", 2024-04-19T02:03:26.178 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.178 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.178 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.178 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.178 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.696711+0000", 2024-04-19T02:03:26.178 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.178 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.178 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.179 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.179 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.696712+0000", 2024-04-19T02:03:26.179 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2799999999999995e-07 2024-04-19T02:03:26.179 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.179 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.179 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.179 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.696725+0000", 2024-04-19T02:03:26.179 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3064000000000001e-05 2024-04-19T02:03:26.179 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.179 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.179 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.179 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.696738+0000", 2024-04-19T02:03:26.179 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3199e-05 2024-04-19T02:03:26.179 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.180 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.180 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.180 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.696748+0000", 2024-04-19T02:03:26.180 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.7850000000000003e-06 2024-04-19T02:03:26.180 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.180 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.180 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.180 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.696828+0000", 2024-04-19T02:03:26.180 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0117999999999994e-05 2024-04-19T02:03:26.180 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.180 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.180 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:26.180 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.696837+0000", 2024-04-19T02:03:26.180 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0569999999999999e-06 2024-04-19T02:03:26.181 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.181 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.181 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:26.182 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.699452+0000", 2024-04-19T02:03:26.182 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0026147589999999999 2024-04-19T02:03:26.182 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.182 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.182 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.182 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.699501+0000", 2024-04-19T02:03:26.182 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9385999999999998e-05 2024-04-19T02:03:26.182 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.182 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.182 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.182 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.183 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.183 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.5s2 35/26 ECSubWrite(tid=31, reqid=client.4509.0:281, at_version=35'13, trim_to=0'0, roll_forward_to=34'12))", 2024-04-19T02:03:26.183 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.740829+0000", 2024-04-19T02:03:26.183 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 6.4132260280000004, 2024-04-19T02:03:26.183 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001782386, 2024-04-19T02:03:26.183 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.183 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.183 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:26.183 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.183 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.183 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.183 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.740829+0000", 2024-04-19T02:03:26.183 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.183 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.184 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.184 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.184 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.740829+0000", 2024-04-19T02:03:26.184 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.184 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.184 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.184 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.184 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.740829+0000", 2024-04-19T02:03:26.184 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.7899999999999999e-07 2024-04-19T02:03:26.184 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.184 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.184 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.184 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.741403+0000", 2024-04-19T02:03:26.185 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000573238 2024-04-19T02:03:26.185 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.185 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.185 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.185 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.741421+0000", 2024-04-19T02:03:26.185 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8953e-05 2024-04-19T02:03:26.185 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.185 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.185 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.185 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.741433+0000", 2024-04-19T02:03:26.185 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1683e-05 2024-04-19T02:03:26.186 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.186 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.186 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.186 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.741505+0000", 2024-04-19T02:03:26.186 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.2028999999999998e-05 2024-04-19T02:03:26.186 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.186 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.186 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:26.186 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.741516+0000", 2024-04-19T02:03:26.186 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0864e-05 2024-04-19T02:03:26.186 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.186 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.186 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:26.186 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.742568+0000", 2024-04-19T02:03:26.186 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0010523869999999999 2024-04-19T02:03:26.187 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.187 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.187 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.187 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.742611+0000", 2024-04-19T02:03:26.187 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2852999999999998e-05 2024-04-19T02:03:26.187 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.187 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.187 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.187 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.187 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.187 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.as7 35/26 ECSubWrite(tid=59, reqid=client.4509.0:285, at_version=35'23, trim_to=0'0, roll_forward_to=34'22))", 2024-04-19T02:03:26.187 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.754512+0000", 2024-04-19T02:03:26.187 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 6.3995427100000004, 2024-04-19T02:03:26.188 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0031998590000000002, 2024-04-19T02:03:26.188 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.188 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.188 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:26.188 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.188 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.188 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.188 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.754512+0000", 2024-04-19T02:03:26.188 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.188 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.188 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.188 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.188 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.754512+0000", 2024-04-19T02:03:26.188 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.189 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.189 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.189 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.189 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.754514+0000", 2024-04-19T02:03:26.189 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.257e-06 2024-04-19T02:03:26.189 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.189 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.189 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.189 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.754796+0000", 2024-04-19T02:03:26.189 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00028269500000000002 2024-04-19T02:03:26.189 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.189 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.189 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.189 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.754814+0000", 2024-04-19T02:03:26.189 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.774e-05 2024-04-19T02:03:26.190 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.190 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.190 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.190 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.754824+0000", 2024-04-19T02:03:26.190 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.8649999999999996e-06 2024-04-19T02:03:26.190 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.190 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.190 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.190 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.754933+0000", 2024-04-19T02:03:26.190 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108769 2024-04-19T02:03:26.190 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.190 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.191 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:26.191 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.754943+0000", 2024-04-19T02:03:26.191 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0071000000000001e-05 2024-04-19T02:03:26.191 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.191 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.191 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:26.191 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.757669+0000", 2024-04-19T02:03:26.191 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0027265929999999998 2024-04-19T02:03:26.191 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.191 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.191 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.191 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.757712+0000", 2024-04-19T02:03:26.191 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2868999999999999e-05 2024-04-19T02:03:26.192 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.192 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.192 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.192 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.192 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.192 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.5s2 37/26 ECSubWrite(tid=38, reqid=client.4509.0:293, at_version=37'17, trim_to=0'0, roll_forward_to=35'15))", 2024-04-19T02:03:26.192 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:21.694800+0000", 2024-04-19T02:03:26.192 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 4.4592544949999997, 2024-04-19T02:03:26.192 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0028351219999999998, 2024-04-19T02:03:26.192 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.192 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.192 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:26.192 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.192 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.192 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.193 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.694800+0000", 2024-04-19T02:03:26.193 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.193 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.193 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.193 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.193 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.694800+0000", 2024-04-19T02:03:26.193 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.193 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.193 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.193 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.193 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.694801+0000", 2024-04-19T02:03:26.193 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.4000000000000001e-07 2024-04-19T02:03:26.193 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.193 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.194 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.194 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.694855+0000", 2024-04-19T02:03:26.194 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.3526999999999998e-05 2024-04-19T02:03:26.194 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.194 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.194 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.194 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.694890+0000", 2024-04-19T02:03:26.194 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5207999999999999e-05 2024-04-19T02:03:26.194 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.194 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.194 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.194 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.694918+0000", 2024-04-19T02:03:26.194 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8657999999999999e-05 2024-04-19T02:03:26.194 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.195 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.195 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.195 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.695183+0000", 2024-04-19T02:03:26.195 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00026409600000000001 2024-04-19T02:03:26.195 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.195 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.195 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:26.195 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.695210+0000", 2024-04-19T02:03:26.195 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7214999999999999e-05 2024-04-19T02:03:26.195 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.195 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.195 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:26.195 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.697535+0000", 2024-04-19T02:03:26.195 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0023248990000000001 2024-04-19T02:03:26.195 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.196 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.196 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.196 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.697636+0000", 2024-04-19T02:03:26.196 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010087899999999999 2024-04-19T02:03:26.196 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.196 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.196 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.196 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.196 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.196 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.5s2 37/26 ECSubWrite(tid=39, reqid=client.4509.0:293, at_version=0'0, trim_to=0'0, roll_forward_to=37'17))", 2024-04-19T02:03:26.196 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:21.700689+0000", 2024-04-19T02:03:26.196 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 4.4533655940000001, 2024-04-19T02:03:26.196 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0016943590000000001, 2024-04-19T02:03:26.196 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.197 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.197 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:26.197 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.197 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.197 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.197 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.700689+0000", 2024-04-19T02:03:26.197 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.197 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.197 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.197 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.197 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.700689+0000", 2024-04-19T02:03:26.197 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.197 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.197 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.197 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.197 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.700691+0000", 2024-04-19T02:03:26.198 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9069999999999999e-06 2024-04-19T02:03:26.198 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.198 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.198 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.198 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.700714+0000", 2024-04-19T02:03:26.198 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3042000000000001e-05 2024-04-19T02:03:26.198 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.198 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.198 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.198 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.700747+0000", 2024-04-19T02:03:26.198 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2804e-05 2024-04-19T02:03:26.198 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.198 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.198 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.198 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.700762+0000", 2024-04-19T02:03:26.199 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5279000000000001e-05 2024-04-19T02:03:26.199 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.199 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.199 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.199 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.700900+0000", 2024-04-19T02:03:26.199 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013753800000000001 2024-04-19T02:03:26.199 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.199 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.199 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:26.199 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.700915+0000", 2024-04-19T02:03:26.199 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4996e-05 2024-04-19T02:03:26.199 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.199 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.199 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:26.199 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.702274+0000", 2024-04-19T02:03:26.199 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0013594499999999999 2024-04-19T02:03:26.200 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.200 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.200 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.200 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:21.702384+0000", 2024-04-19T02:03:26.200 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000109343 2024-04-19T02:03:26.200 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.200 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.200 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.200 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.200 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.200 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s5 40/26 ECSubWrite(tid=37, reqid=client.4509.0:344, at_version=40'25, trim_to=0'0, roll_forward_to=39'23))", 2024-04-19T02:03:26.200 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.739704+0000", 2024-04-19T02:03:26.200 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.4143507710000001, 2024-04-19T02:03:26.200 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0020610009999999998, 2024-04-19T02:03:26.200 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.201 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.201 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:26.201 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.201 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.201 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.201 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.739704+0000", 2024-04-19T02:03:26.201 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.201 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.201 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.201 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.201 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.739704+0000", 2024-04-19T02:03:26.201 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.201 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.201 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.201 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.202 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.739705+0000", 2024-04-19T02:03:26.202 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.01e-07 2024-04-19T02:03:26.202 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.202 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.202 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.202 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.739831+0000", 2024-04-19T02:03:26.202 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000126549 2024-04-19T02:03:26.202 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.202 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.202 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.202 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.739846+0000", 2024-04-19T02:03:26.202 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4854999999999999e-05 2024-04-19T02:03:26.202 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.202 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.202 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.203 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.739858+0000", 2024-04-19T02:03:26.203 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1948e-05 2024-04-19T02:03:26.203 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.203 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.203 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.203 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.739967+0000", 2024-04-19T02:03:26.203 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010887699999999999 2024-04-19T02:03:26.203 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.203 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.203 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:26.203 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.739974+0000", 2024-04-19T02:03:26.203 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6350000000000006e-06 2024-04-19T02:03:26.203 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.203 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.203 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:26.204 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.741733+0000", 2024-04-19T02:03:26.204 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001758082 2024-04-19T02:03:26.204 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.204 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.204 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.204 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.741765+0000", 2024-04-19T02:03:26.204 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2654000000000003e-05 2024-04-19T02:03:26.204 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.204 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.204 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.204 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.204 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.204 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s5 40/26 ECSubWrite(tid=38, reqid=client.4509.0:345, at_version=40'26, trim_to=0'0, roll_forward_to=39'23))", 2024-04-19T02:03:26.204 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.747958+0000", 2024-04-19T02:03:26.204 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.4060969700000001, 2024-04-19T02:03:26.204 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0030891349999999998, 2024-04-19T02:03:26.205 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.205 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.205 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:26.205 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.205 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.205 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.205 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.747958+0000", 2024-04-19T02:03:26.205 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.205 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.205 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.205 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.206 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.747958+0000", 2024-04-19T02:03:26.206 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.206 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.206 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.206 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.206 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.747960+0000", 2024-04-19T02:03:26.206 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.699e-06 2024-04-19T02:03:26.206 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.206 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.206 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.206 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.748180+0000", 2024-04-19T02:03:26.206 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00021990700000000001 2024-04-19T02:03:26.206 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.206 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.206 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.207 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.748219+0000", 2024-04-19T02:03:26.207 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9447999999999999e-05 2024-04-19T02:03:26.207 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.207 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.207 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.207 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.748237+0000", 2024-04-19T02:03:26.207 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8036000000000001e-05 2024-04-19T02:03:26.207 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.207 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.207 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.207 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.748317+0000", 2024-04-19T02:03:26.208 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0060000000000003e-05 2024-04-19T02:03:26.208 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.208 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.208 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:26.208 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.748328+0000", 2024-04-19T02:03:26.208 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.131e-05 2024-04-19T02:03:26.208 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.208 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.208 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:26.208 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750994+0000", 2024-04-19T02:03:26.208 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002665541 2024-04-19T02:03:26.209 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.209 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.209 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.209 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.751047+0000", 2024-04-19T02:03:26.209 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.3134000000000003e-05 2024-04-19T02:03:26.209 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.209 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.209 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.209 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.209 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.209 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s5 40/26 ECSubWrite(tid=39, reqid=client.4509.0:346, at_version=40'27, trim_to=0'0, roll_forward_to=39'23))", 2024-04-19T02:03:26.209 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.749260+0000", 2024-04-19T02:03:26.209 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.404795046, 2024-04-19T02:03:26.209 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0024379459999999999, 2024-04-19T02:03:26.210 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.210 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.210 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:26.210 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.210 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.210 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.210 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749260+0000", 2024-04-19T02:03:26.210 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.210 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.210 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.210 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.211 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749260+0000", 2024-04-19T02:03:26.211 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.211 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.211 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.211 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.211 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749261+0000", 2024-04-19T02:03:26.211 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.08e-06 2024-04-19T02:03:26.211 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.211 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.211 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.211 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749321+0000", 2024-04-19T02:03:26.212 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.995e-05 2024-04-19T02:03:26.212 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.212 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.212 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.212 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749343+0000", 2024-04-19T02:03:26.212 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1817000000000001e-05 2024-04-19T02:03:26.212 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.212 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.212 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.212 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749357+0000", 2024-04-19T02:03:26.212 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4093999999999999e-05 2024-04-19T02:03:26.212 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.212 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.212 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.212 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750142+0000", 2024-04-19T02:03:26.213 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00078503399999999999 2024-04-19T02:03:26.213 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.213 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.213 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:26.213 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750160+0000", 2024-04-19T02:03:26.213 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8009999999999999e-05 2024-04-19T02:03:26.213 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.213 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.213 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:26.213 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.751644+0000", 2024-04-19T02:03:26.213 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0014845080000000001 2024-04-19T02:03:26.213 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.214 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.214 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.214 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.751698+0000", 2024-04-19T02:03:26.214 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.3452999999999998e-05 2024-04-19T02:03:26.214 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.214 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.214 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.214 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.214 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.214 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es5 40/26 ECSubWrite(tid=26, reqid=client.4509.0:351, at_version=40'20, trim_to=0'0, roll_forward_to=35'18))", 2024-04-19T02:03:26.214 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.765179+0000", 2024-04-19T02:03:26.214 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.3888759880000001, 2024-04-19T02:03:26.214 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001768758, 2024-04-19T02:03:26.214 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.215 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.215 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:26.215 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.215 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.215 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.215 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.765179+0000", 2024-04-19T02:03:26.215 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.215 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.215 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.215 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.215 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.765179+0000", 2024-04-19T02:03:26.215 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.215 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.215 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.215 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.216 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.765180+0000", 2024-04-19T02:03:26.216 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.7899999999999998e-07 2024-04-19T02:03:26.216 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.216 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.216 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.216 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.765659+0000", 2024-04-19T02:03:26.216 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00047959700000000001 2024-04-19T02:03:26.216 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.216 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.216 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.216 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.765674+0000", 2024-04-19T02:03:26.216 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5218999999999999e-05 2024-04-19T02:03:26.216 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.216 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.216 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.217 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.765685+0000", 2024-04-19T02:03:26.217 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0316000000000001e-05 2024-04-19T02:03:26.217 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.217 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.217 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.217 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.765752+0000", 2024-04-19T02:03:26.217 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.7516999999999997e-05 2024-04-19T02:03:26.217 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.217 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.217 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:26.217 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.765763+0000", 2024-04-19T02:03:26.217 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0232e-05 2024-04-19T02:03:26.217 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.217 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.217 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:26.217 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.766901+0000", 2024-04-19T02:03:26.218 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0011385239999999999 2024-04-19T02:03:26.218 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.218 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.218 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.218 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.766948+0000", 2024-04-19T02:03:26.218 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.6674000000000001e-05 2024-04-19T02:03:26.218 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.218 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.218 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.218 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.218 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.218 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es5 40/26 ECSubWrite(tid=27, reqid=client.4509.0:352, at_version=40'21, trim_to=0'0, roll_forward_to=35'18))", 2024-04-19T02:03:26.218 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.771783+0000", 2024-04-19T02:03:26.218 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.3822721339999999, 2024-04-19T02:03:26.218 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0015981509999999999, 2024-04-19T02:03:26.219 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.219 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.219 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:26.219 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.219 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.219 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.219 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.771783+0000", 2024-04-19T02:03:26.219 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.219 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.219 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.219 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.219 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.771783+0000", 2024-04-19T02:03:26.219 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.219 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.219 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.220 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.220 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.771784+0000", 2024-04-19T02:03:26.220 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2920000000000001e-06 2024-04-19T02:03:26.220 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.220 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.220 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.220 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772073+0000", 2024-04-19T02:03:26.220 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00028904700000000001 2024-04-19T02:03:26.220 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.220 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.220 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.220 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772103+0000", 2024-04-19T02:03:26.220 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9725000000000001e-05 2024-04-19T02:03:26.220 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.220 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.220 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.221 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772117+0000", 2024-04-19T02:03:26.221 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4229e-05 2024-04-19T02:03:26.221 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.221 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.221 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.221 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772232+0000", 2024-04-19T02:03:26.221 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00011516299999999999 2024-04-19T02:03:26.221 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.221 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.221 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:26.221 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772252+0000", 2024-04-19T02:03:26.221 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9256e-05 2024-04-19T02:03:26.221 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.221 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.221 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:26.222 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.773334+0000", 2024-04-19T02:03:26.222 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0010826410000000001 2024-04-19T02:03:26.222 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.222 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.222 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.222 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.773381+0000", 2024-04-19T02:03:26.222 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.6798000000000002e-05 2024-04-19T02:03:26.222 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.222 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.222 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.222 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.222 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.222 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es5 40/26 ECSubWrite(tid=28, reqid=client.4509.0:353, at_version=40'22, trim_to=0'0, roll_forward_to=40'20))", 2024-04-19T02:03:26.222 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.776809+0000", 2024-04-19T02:03:26.222 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.377245718, 2024-04-19T02:03:26.223 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001950151, 2024-04-19T02:03:26.223 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.223 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.223 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:26.223 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.223 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.223 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.223 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.776809+0000", 2024-04-19T02:03:26.223 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.223 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.223 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.223 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.223 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.776809+0000", 2024-04-19T02:03:26.223 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.223 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.224 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.224 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.224 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.776810+0000", 2024-04-19T02:03:26.224 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.11e-06 2024-04-19T02:03:26.224 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.224 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.224 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.224 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.776956+0000", 2024-04-19T02:03:26.224 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000145697 2024-04-19T02:03:26.224 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.224 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.224 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.224 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.776976+0000", 2024-04-19T02:03:26.224 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9738e-05 2024-04-19T02:03:26.224 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.225 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.225 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.225 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.776987+0000", 2024-04-19T02:03:26.225 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1547e-05 2024-04-19T02:03:26.225 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.225 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.225 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.225 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.777514+0000", 2024-04-19T02:03:26.225 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00052618599999999999 2024-04-19T02:03:26.225 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.225 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.225 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:26.225 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.777529+0000", 2024-04-19T02:03:26.225 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5243e-05 2024-04-19T02:03:26.225 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.225 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.226 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:26.226 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.778623+0000", 2024-04-19T02:03:26.226 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0010940590000000001 2024-04-19T02:03:26.226 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.226 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.226 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.226 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.778759+0000", 2024-04-19T02:03:26.226 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013657099999999999 2024-04-19T02:03:26.226 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.226 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.226 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.226 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.226 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.226 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es5 40/26 ECSubWrite(tid=29, reqid=client.4509.0:354, at_version=40'23, trim_to=0'0, roll_forward_to=40'20))", 2024-04-19T02:03:26.227 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.779583+0000", 2024-04-19T02:03:26.227 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.3744723910000001, 2024-04-19T02:03:26.227 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0015890360000000001, 2024-04-19T02:03:26.227 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.227 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.227 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:26.227 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.227 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.227 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.227 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.779583+0000", 2024-04-19T02:03:26.227 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.227 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.227 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.227 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.228 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.779583+0000", 2024-04-19T02:03:26.228 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.228 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.228 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.228 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.228 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.779584+0000", 2024-04-19T02:03:26.228 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0809999999999999e-06 2024-04-19T02:03:26.228 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.228 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.228 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.228 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.779605+0000", 2024-04-19T02:03:26.228 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1350000000000001e-05 2024-04-19T02:03:26.228 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.228 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.228 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.229 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.779623+0000", 2024-04-19T02:03:26.229 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7708000000000001e-05 2024-04-19T02:03:26.229 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.229 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.229 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.229 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.779633+0000", 2024-04-19T02:03:26.229 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0559e-05 2024-04-19T02:03:26.229 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.229 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.229 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.229 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.779742+0000", 2024-04-19T02:03:26.229 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108684 2024-04-19T02:03:26.229 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.229 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.229 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:26.230 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.779755+0000", 2024-04-19T02:03:26.230 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3142999999999999e-05 2024-04-19T02:03:26.230 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.230 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.230 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:26.230 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.781122+0000", 2024-04-19T02:03:26.230 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0013665439999999999 2024-04-19T02:03:26.230 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.230 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.230 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.230 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.781172+0000", 2024-04-19T02:03:26.230 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9966999999999998e-05 2024-04-19T02:03:26.230 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.231 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.231 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.231 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.231 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.231 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es5 40/26 ECSubWrite(tid=30, reqid=client.4509.0:354, at_version=0'0, trim_to=0'0, roll_forward_to=40'23))", 2024-04-19T02:03:26.231 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.784038+0000", 2024-04-19T02:03:26.231 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.370016892, 2024-04-19T02:03:26.231 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.003465409, 2024-04-19T02:03:26.231 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.231 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.231 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:26.231 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.231 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.231 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.232 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.784038+0000", 2024-04-19T02:03:26.232 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.232 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.232 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.232 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.232 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.784038+0000", 2024-04-19T02:03:26.232 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.232 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.232 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.232 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.232 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.784039+0000", 2024-04-19T02:03:26.232 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.032e-06 2024-04-19T02:03:26.232 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.232 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.232 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.233 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.784052+0000", 2024-04-19T02:03:26.233 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3366e-05 2024-04-19T02:03:26.233 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.233 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.233 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.233 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.784074+0000", 2024-04-19T02:03:26.233 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1571999999999999e-05 2024-04-19T02:03:26.233 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.233 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.233 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.233 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.784083+0000", 2024-04-19T02:03:26.233 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3910000000000004e-06 2024-04-19T02:03:26.233 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.234 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.234 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.234 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.784159+0000", 2024-04-19T02:03:26.234 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6032000000000007e-05 2024-04-19T02:03:26.234 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.234 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.234 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:26.234 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.784171+0000", 2024-04-19T02:03:26.234 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1839e-05 2024-04-19T02:03:26.234 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.234 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.234 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:26.234 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.787472+0000", 2024-04-19T02:03:26.235 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0033007969999999998 2024-04-19T02:03:26.235 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.235 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.235 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.235 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.787503+0000", 2024-04-19T02:03:26.235 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1380000000000001e-05 2024-04-19T02:03:26.235 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.235 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.235 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.235 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.235 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.235 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 40/26 ECSubWrite(tid=30, reqid=client.4509.0:359, at_version=40'16, trim_to=0'0, roll_forward_to=39'14))", 2024-04-19T02:03:26.235 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.790646+0000", 2024-04-19T02:03:26.235 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.363408478, 2024-04-19T02:03:26.235 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.004294422, 2024-04-19T02:03:26.236 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.236 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.236 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:26.236 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.236 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.236 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.236 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.790646+0000", 2024-04-19T02:03:26.236 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.236 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.236 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.236 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.236 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.790646+0000", 2024-04-19T02:03:26.236 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.236 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.236 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.236 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.237 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.790647+0000", 2024-04-19T02:03:26.237 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7400000000000003e-07 2024-04-19T02:03:26.237 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.237 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.237 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.237 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.791098+0000", 2024-04-19T02:03:26.237 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00045124599999999998 2024-04-19T02:03:26.237 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.237 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.237 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.237 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.791116+0000", 2024-04-19T02:03:26.237 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7351999999999999e-05 2024-04-19T02:03:26.237 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.237 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.237 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.238 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.791126+0000", 2024-04-19T02:03:26.238 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0606e-05 2024-04-19T02:03:26.238 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.238 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.238 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.238 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.791206+0000", 2024-04-19T02:03:26.238 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0018999999999994e-05 2024-04-19T02:03:26.238 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.238 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.238 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:26.238 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.791217+0000", 2024-04-19T02:03:26.238 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0361e-05 2024-04-19T02:03:26.238 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.238 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.238 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:26.239 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.794897+0000", 2024-04-19T02:03:26.239 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0036802810000000001 2024-04-19T02:03:26.239 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.239 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.239 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.239 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.794941+0000", 2024-04-19T02:03:26.239 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.3983000000000002e-05 2024-04-19T02:03:26.239 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.239 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.239 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.239 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.239 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.239 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:26.241 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-19T02:03:26.241 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:26.239+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:26.342 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:26.342+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:26.414 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:26.414 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:03:26.414 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-04-19T02:03:26.414 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:26.429 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-19T02:03:26.443 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:26.447+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:26.544 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:26.543+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:26.549 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:26.549 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:03:26.549 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-04-19T02:03:26.549 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-04-19T02:03:26.549 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:26.563 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-19T02:03:26.645 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:26.643+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:26.687 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:26.687 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-04-19T02:03:26.687 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-04-19T02:03:26.687 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-04-19T02:03:26.687 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.687 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:230 3.2s0 3:43ce42f3:::smithi05930959-27:head [stat,delete] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-19T02:03:26.687 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:15.635171+0000", 2024-04-19T02:03:26.687 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 11.052188706000001, 2024-04-19T02:03:26.687 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.013910249, 2024-04-19T02:03:26.687 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.688 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.688 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:26.688 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:26.688 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:26.688 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:26.688 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 230 2024-04-19T02:03:26.688 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.688 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.688 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.688 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.688 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.635171+0000", 2024-04-19T02:03:26.688 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.688 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.689 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.689 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.689 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.635171+0000", 2024-04-19T02:03:26.689 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.689 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.689 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.689 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.689 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.635175+0000", 2024-04-19T02:03:26.689 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2399999999999999e-06 2024-04-19T02:03:26.689 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.689 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.689 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.689 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.635190+0000", 2024-04-19T02:03:26.689 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.517e-05 2024-04-19T02:03:26.690 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.690 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.690 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.690 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.635226+0000", 2024-04-19T02:03:26.690 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6495999999999998e-05 2024-04-19T02:03:26.690 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.690 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.690 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.690 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.635250+0000", 2024-04-19T02:03:26.690 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3713000000000001e-05 2024-04-19T02:03:26.690 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.690 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.690 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.690 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.635429+0000", 2024-04-19T02:03:26.691 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00017871599999999999 2024-04-19T02:03:26.691 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.691 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.691 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "op must wait for map", 2024-04-19T02:03:26.691 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.635456+0000", 2024-04-19T02:03:26.691 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7373e-05 2024-04-19T02:03:26.691 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.691 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.691 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.691 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.639506+0000", 2024-04-19T02:03:26.691 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0040497620000000002 2024-04-19T02:03:26.691 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.691 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.691 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:26.692 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.639758+0000", 2024-04-19T02:03:26.692 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00025186499999999999 2024-04-19T02:03:26.692 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.692 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.692 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:26.692 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.641987+0000", 2024-04-19T02:03:26.692 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0022297290000000002 2024-04-19T02:03:26.692 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.692 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.692 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:26.692 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.644866+0000", 2024-04-19T02:03:26.692 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0028787550000000002 2024-04-19T02:03:26.692 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.692 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.693 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:26.693 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.648961+0000", 2024-04-19T02:03:26.693 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0040949979999999999 2024-04-19T02:03:26.693 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.693 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.693 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.693 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.649082+0000", 2024-04-19T02:03:26.693 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000120432 2024-04-19T02:03:26.693 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.693 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.693 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.693 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.693 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.693 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:231 3.2s0 3:439cb4d1:::smithi05930959-23:head [append 0~671744] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-19T02:03:26.693 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:15.657794+0000", 2024-04-19T02:03:26.694 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 11.029566407000001, 2024-04-19T02:03:26.694 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012024274, 2024-04-19T02:03:26.694 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.694 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.694 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:26.694 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:26.694 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:26.694 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:26.694 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 231 2024-04-19T02:03:26.694 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.694 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.694 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.694 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.694 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.657794+0000", 2024-04-19T02:03:26.695 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.695 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.695 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.695 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.695 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.657794+0000", 2024-04-19T02:03:26.695 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.695 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.695 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.695 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.695 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.657797+0000", 2024-04-19T02:03:26.695 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.23e-06 2024-04-19T02:03:26.695 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.695 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.695 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.695 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.658419+0000", 2024-04-19T02:03:26.696 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00062175499999999999 2024-04-19T02:03:26.696 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.696 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.696 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.696 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.658453+0000", 2024-04-19T02:03:26.696 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4116999999999999e-05 2024-04-19T02:03:26.696 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.696 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.696 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.696 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.658475+0000", 2024-04-19T02:03:26.696 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2203000000000001e-05 2024-04-19T02:03:26.696 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.696 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.696 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.696 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.658611+0000", 2024-04-19T02:03:26.697 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000136425 2024-04-19T02:03:26.697 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.697 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.697 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:26.697 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.658797+0000", 2024-04-19T02:03:26.697 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00018583599999999999 2024-04-19T02:03:26.697 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.697 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.697 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:26.697 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661233+0000", 2024-04-19T02:03:26.697 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0024353869999999998 2024-04-19T02:03:26.697 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.697 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.697 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:26.698 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.668947+0000", 2024-04-19T02:03:26.698 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0077140139999999999 2024-04-19T02:03:26.698 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.698 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.698 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:26.698 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.669636+0000", 2024-04-19T02:03:26.698 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00068981399999999997 2024-04-19T02:03:26.698 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.698 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.698 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.698 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.669818+0000", 2024-04-19T02:03:26.698 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00018149299999999999 2024-04-19T02:03:26.698 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.698 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.699 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.699 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.699 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.699 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:232 3.2s0 3:439cb4d1:::smithi05930959-23:head [append 0~376832] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-19T02:03:26.699 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:15.660690+0000", 2024-04-19T02:03:26.699 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 11.026669553, 2024-04-19T02:03:26.699 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010153852, 2024-04-19T02:03:26.699 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.699 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.699 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:26.699 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:26.699 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:26.699 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:26.699 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 232 2024-04-19T02:03:26.699 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.700 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.700 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.700 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.700 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.660690+0000", 2024-04-19T02:03:26.700 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.700 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.700 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.700 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.700 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.660690+0000", 2024-04-19T02:03:26.700 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.700 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.700 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.700 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.701 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.660693+0000", 2024-04-19T02:03:26.701 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.903e-06 2024-04-19T02:03:26.701 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.701 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.701 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.701 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661327+0000", 2024-04-19T02:03:26.701 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00063362200000000003 2024-04-19T02:03:26.701 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.701 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.701 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.701 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661402+0000", 2024-04-19T02:03:26.701 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.5190000000000003e-05 2024-04-19T02:03:26.701 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.701 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.702 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.702 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661426+0000", 2024-04-19T02:03:26.702 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3519e-05 2024-04-19T02:03:26.702 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.702 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.702 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.702 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.663292+0000", 2024-04-19T02:03:26.702 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0018659989999999999 2024-04-19T02:03:26.702 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.702 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.702 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:26.702 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.663512+0000", 2024-04-19T02:03:26.702 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000220118 2024-04-19T02:03:26.702 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.703 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.703 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:26.703 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.665317+0000", 2024-04-19T02:03:26.703 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0018056260000000001 2024-04-19T02:03:26.703 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.703 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.703 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:26.703 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.668949+0000", 2024-04-19T02:03:26.703 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.003631563 2024-04-19T02:03:26.703 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.703 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.703 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:26.703 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.670735+0000", 2024-04-19T02:03:26.703 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001786096 2024-04-19T02:03:26.703 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.704 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.704 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.704 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.670844+0000", 2024-04-19T02:03:26.704 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010921600000000001 2024-04-19T02:03:26.704 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.704 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.704 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.704 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.704 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.704 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:233 3.2s0 3:439cb4d1:::smithi05930959-23:head [setxattr _header (56) in=7b] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-19T02:03:26.704 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:15.661479+0000", 2024-04-19T02:03:26.704 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 11.025880879000001, 2024-04-19T02:03:26.704 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010065869999999999, 2024-04-19T02:03:26.704 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.704 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.704 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:26.705 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:26.705 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:26.705 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:26.705 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 233 2024-04-19T02:03:26.705 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.705 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.705 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.705 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.705 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661479+0000", 2024-04-19T02:03:26.705 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.705 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.705 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.705 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.705 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661479+0000", 2024-04-19T02:03:26.705 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.705 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.705 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.706 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.706 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661481+0000", 2024-04-19T02:03:26.706 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.72e-06 2024-04-19T02:03:26.706 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.706 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.706 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.706 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661493+0000", 2024-04-19T02:03:26.706 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1653e-05 2024-04-19T02:03:26.706 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.706 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.706 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.706 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661517+0000", 2024-04-19T02:03:26.706 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4749999999999999e-05 2024-04-19T02:03:26.706 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.706 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.707 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.707 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661535+0000", 2024-04-19T02:03:26.707 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8003e-05 2024-04-19T02:03:26.707 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.707 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.707 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.707 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.666609+0000", 2024-04-19T02:03:26.707 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.005073691 2024-04-19T02:03:26.707 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.707 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.707 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:26.707 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.666880+0000", 2024-04-19T02:03:26.707 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00027123099999999998 2024-04-19T02:03:26.707 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.708 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.708 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:26.708 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.668034+0000", 2024-04-19T02:03:26.708 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001153421 2024-04-19T02:03:26.708 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.708 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.708 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:26.708 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.670252+0000", 2024-04-19T02:03:26.708 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002218397 2024-04-19T02:03:26.708 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.708 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.708 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:26.708 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.671448+0000", 2024-04-19T02:03:26.708 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0011963189999999999 2024-04-19T02:03:26.708 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.708 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.709 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.709 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.671545+0000", 2024-04-19T02:03:26.709 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.6685e-05 2024-04-19T02:03:26.709 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.709 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.709 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.709 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.709 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.709 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:234 3.2s0 3:439cb4d1:::smithi05930959-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-19T02:03:26.709 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:15.661605+0000", 2024-04-19T02:03:26.709 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 11.025754593, 2024-04-19T02:03:26.709 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.013322123, 2024-04-19T02:03:26.709 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.710 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.710 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:26.710 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:26.710 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:26.710 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:26.710 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 234 2024-04-19T02:03:26.710 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.710 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.710 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.710 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.710 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661605+0000", 2024-04-19T02:03:26.710 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.710 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.710 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.711 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.711 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661605+0000", 2024-04-19T02:03:26.711 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.711 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.711 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.711 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.711 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661607+0000", 2024-04-19T02:03:26.711 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5600000000000001e-06 2024-04-19T02:03:26.711 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.711 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.711 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.711 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661617+0000", 2024-04-19T02:03:26.711 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0436e-05 2024-04-19T02:03:26.711 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.712 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.712 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.712 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661644+0000", 2024-04-19T02:03:26.712 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6376999999999998e-05 2024-04-19T02:03:26.712 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.712 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.712 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.712 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.661662+0000", 2024-04-19T02:03:26.712 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7710000000000002e-05 2024-04-19T02:03:26.712 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.712 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.712 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.712 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.672340+0000", 2024-04-19T02:03:26.712 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010678974000000001 2024-04-19T02:03:26.712 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.713 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.713 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:26.713 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.672598+0000", 2024-04-19T02:03:26.713 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00025784799999999997 2024-04-19T02:03:26.713 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.713 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.713 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.713 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:15.674928+0000", 2024-04-19T02:03:26.713 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0023292180000000001 2024-04-19T02:03:26.713 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.713 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.713 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.713 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.713 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.713 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:236 3.2s0 3:4aa53487:::smithi05930959-17:head [create,setxattr pUpejN\\XeR[ScpqbhnaXajbqbVcs]\\YUaoVqmdSQ (16807) in=40b,setxattr ^TXcXY\\i\\UaegqjXsVZRnYhXQcQtNan^ZkmbhgfY (349) in=40b,setxattr ekSnl]Q^sZQXVqUlaf`k_aifQmosa`q\\TjqRt_Qf (1227) in=40b,setxattr \\iWlX_lYknVg`YeraZ]fTRtp[SmcmU\\fi`W]WrbQ (6244) in=40b,setxattr jXafahNhrkaS]`RTUfZsViWZQb_fe[hNa`eaSXaP (104) in=40b,setxattr SQfdcs\\p[W^UXTPlNX^ZpfRc`f[oTemUqrW]noj\\ (1227) in=40b,setxattr \\^nWTlXbgpUbdWkgbnaQn`^hmQ[]WmV]lU[SaXU] (1542) in=40b,setxattr dVRspsbTY\\[_ajteqtXtW[`oNOrUtVnqhPan_ePQ (314) in=40b,setxattr tQcXlPoWcYqTci^srZesYV[VgVaRshSrZtc\\mZa^ (67) in=40b,setxattr s]QeT_cN\\hZPf_Whl\\VZhhldY`gN^hNQ[rbdZ\\aZ (244) in=40b,setxattr Sga\\QYQOYspmkSbrWjWZXUeqopRefVUPXn]SjYcY (1719) in=40b,setxattr [NiqOQcWdfT`oVaqXYf_dm`h_g]OscjStsShYaUN (876) in=40b,setxattr Ok_\\_emNX_rOSq`pl[ijceSO_WNtXVhjhtTPolkN (1647) in=40b,setxattr saRsYUrZhdhmY^k^i[Rph][cqVtPsi`WarjOnYO^ (1155) in=40b,setxattr cY\\acssfT[QaU\\pigijbXs\\ROcY__s[R_geknqaa (1472) in=40b,setxattr WfSmRUdrW[ojWRr_kTre\\ns`t[cOrTVTNankd[OP (350) in=40b,setxattr T`ce]ZdmspasZU]X[c[socq[elaihoTSQi_WieWt (1226) in=40b,setxattr piRXhWpPba^f_P_Z[\\`\\Ugj[kRno\\VgbWQlndVa] (1718) in=40b,setxattr lV]d\\p[Pbb_ShS]kXZmPXqpq`QhRa`cXeNpTNciq (875) in=40b,setxattr cdtSkTmQRaR]lfSnj]qi[pkO`[NRbaqUWsnj]XiO (314) in=40b] snapc 3=[2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-19T02:03:26.714 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.646243+0000", 2024-04-19T02:03:26.714 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 10.041117218, 2024-04-19T02:03:26.714 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.022457007000000001, 2024-04-19T02:03:26.714 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.714 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.714 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:26.714 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:26.714 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:26.714 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:26.714 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 236 2024-04-19T02:03:26.714 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.714 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.714 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.714 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.715 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646243+0000", 2024-04-19T02:03:26.715 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.715 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.715 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.715 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.715 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646243+0000", 2024-04-19T02:03:26.715 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.715 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.715 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.715 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.715 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646246+0000", 2024-04-19T02:03:26.715 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.92e-06 2024-04-19T02:03:26.715 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.715 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.715 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.716 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646351+0000", 2024-04-19T02:03:26.716 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000105393 2024-04-19T02:03:26.716 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.716 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.716 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.716 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646385+0000", 2024-04-19T02:03:26.716 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4326000000000003e-05 2024-04-19T02:03:26.716 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.716 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.716 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.716 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646403+0000", 2024-04-19T02:03:26.716 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7543000000000001e-05 2024-04-19T02:03:26.716 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.716 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.716 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.717 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646534+0000", 2024-04-19T02:03:26.717 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013077600000000001 2024-04-19T02:03:26.717 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.717 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.717 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "op must wait for map", 2024-04-19T02:03:26.717 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646600+0000", 2024-04-19T02:03:26.717 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.5977999999999997e-05 2024-04-19T02:03:26.717 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.717 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.717 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.717 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.650956+0000", 2024-04-19T02:03:26.717 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0043565890000000001 2024-04-19T02:03:26.717 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.717 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.717 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:26.717 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.651213+0000", 2024-04-19T02:03:26.718 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00025659899999999999 2024-04-19T02:03:26.718 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.718 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.718 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:26.718 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.655246+0000", 2024-04-19T02:03:26.718 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.004033549 2024-04-19T02:03:26.718 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.718 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.718 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:26.718 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.668007+0000", 2024-04-19T02:03:26.718 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012760535 2024-04-19T02:03:26.718 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.718 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.718 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:26.719 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.668552+0000", 2024-04-19T02:03:26.719 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00054500000000000002 2024-04-19T02:03:26.719 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.719 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.719 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.719 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.668700+0000", 2024-04-19T02:03:26.719 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000147799 2024-04-19T02:03:26.719 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.719 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.719 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.719 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.719 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.719 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:237 3.2s0 3:407ecba7:::smithi05930959-22:2 [read 0~360448,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-19T02:03:26.719 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.646438+0000", 2024-04-19T02:03:26.719 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 10.040921560999999, 2024-04-19T02:03:26.720 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.024327603, 2024-04-19T02:03:26.720 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.720 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.720 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:26.720 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:26.720 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:26.720 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:26.720 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 237 2024-04-19T02:03:26.720 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.720 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.720 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.720 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.720 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646438+0000", 2024-04-19T02:03:26.720 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.720 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.720 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.721 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.721 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646438+0000", 2024-04-19T02:03:26.721 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.721 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.721 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.721 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.721 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646440+0000", 2024-04-19T02:03:26.721 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.254e-06 2024-04-19T02:03:26.721 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.721 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.721 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.721 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646447+0000", 2024-04-19T02:03:26.721 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.3289999999999996e-06 2024-04-19T02:03:26.721 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.721 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.722 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.722 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646463+0000", 2024-04-19T02:03:26.722 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5846e-05 2024-04-19T02:03:26.722 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.722 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.722 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.722 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646472+0000", 2024-04-19T02:03:26.722 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.2939999999999994e-06 2024-04-19T02:03:26.722 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.722 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.722 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.722 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646953+0000", 2024-04-19T02:03:26.722 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000480901 2024-04-19T02:03:26.722 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.723 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.723 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:03:26.723 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646971+0000", 2024-04-19T02:03:26.723 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8167000000000001e-05 2024-04-19T02:03:26.723 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.723 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.723 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.723 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.656667+0000", 2024-04-19T02:03:26.723 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0096952449999999999 2024-04-19T02:03:26.723 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.723 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.723 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:26.723 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.656833+0000", 2024-04-19T02:03:26.723 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00016697500000000001 2024-04-19T02:03:26.723 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.724 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.724 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.724 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.670766+0000", 2024-04-19T02:03:26.724 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.013932592000000001 2024-04-19T02:03:26.724 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.724 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.724 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.724 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.724 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.724 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:238 3.2s0 3:407ecba7:::smithi05930959-22:2 [read 0~360448,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-19T02:03:26.724 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.646510+0000", 2024-04-19T02:03:26.724 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 10.040850279000001, 2024-04-19T02:03:26.724 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.028744658999999999, 2024-04-19T02:03:26.724 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.725 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.725 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:26.725 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:26.725 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:26.725 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:26.725 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 238 2024-04-19T02:03:26.725 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.725 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.725 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.725 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.725 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646510+0000", 2024-04-19T02:03:26.725 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.725 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.725 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.726 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.726 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646510+0000", 2024-04-19T02:03:26.726 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.726 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.726 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.726 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.726 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646511+0000", 2024-04-19T02:03:26.726 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.8599999999999997e-07 2024-04-19T02:03:26.726 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.726 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.726 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.726 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646517+0000", 2024-04-19T02:03:26.726 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.1700000000000002e-06 2024-04-19T02:03:26.727 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.727 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.727 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.727 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646528+0000", 2024-04-19T02:03:26.727 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1202999999999999e-05 2024-04-19T02:03:26.727 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.727 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.727 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.727 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646537+0000", 2024-04-19T02:03:26.727 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7460000000000006e-06 2024-04-19T02:03:26.727 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.728 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.728 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.728 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.647105+0000", 2024-04-19T02:03:26.728 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00056801599999999996 2024-04-19T02:03:26.728 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.728 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.728 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:03:26.728 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.647127+0000", 2024-04-19T02:03:26.728 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.175e-05 2024-04-19T02:03:26.728 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.728 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.728 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.728 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.657504+0000", 2024-04-19T02:03:26.728 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010377961 2024-04-19T02:03:26.729 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.729 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.729 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:26.729 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.657826+0000", 2024-04-19T02:03:26.729 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00032128300000000001 2024-04-19T02:03:26.729 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.729 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.729 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.729 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.675254+0000", 2024-04-19T02:03:26.729 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.017428644 2024-04-19T02:03:26.729 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.729 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.729 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.729 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.730 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.730 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:239 3.2s0 3:407ecba7:::smithi05930959-22:2 [read 0~360448,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-19T02:03:26.730 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.646549+0000", 2024-04-19T02:03:26.730 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 10.040810808, 2024-04-19T02:03:26.730 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.029381570999999999, 2024-04-19T02:03:26.730 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.730 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.730 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:26.730 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:26.730 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:26.730 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:26.730 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 239 2024-04-19T02:03:26.730 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.730 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.730 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.731 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.731 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646549+0000", 2024-04-19T02:03:26.731 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.731 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.731 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.731 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.731 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646549+0000", 2024-04-19T02:03:26.731 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.731 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.731 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.731 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.731 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646550+0000", 2024-04-19T02:03:26.731 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8400000000000004e-07 2024-04-19T02:03:26.731 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.731 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.731 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.732 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646555+0000", 2024-04-19T02:03:26.732 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9989999999999999e-06 2024-04-19T02:03:26.732 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.732 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.732 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.732 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646567+0000", 2024-04-19T02:03:26.732 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.202e-05 2024-04-19T02:03:26.732 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.732 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.732 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.732 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646575+0000", 2024-04-19T02:03:26.732 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.7640000000000005e-06 2024-04-19T02:03:26.732 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.732 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.732 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.733 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.647229+0000", 2024-04-19T02:03:26.733 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00065415299999999998 2024-04-19T02:03:26.733 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.733 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.733 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:03:26.733 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.647255+0000", 2024-04-19T02:03:26.733 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5795999999999998e-05 2024-04-19T02:03:26.733 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.733 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.733 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.733 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.659835+0000", 2024-04-19T02:03:26.733 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012580713 2024-04-19T02:03:26.733 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.733 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.733 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:26.734 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.660090+0000", 2024-04-19T02:03:26.734 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00025430899999999998 2024-04-19T02:03:26.734 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.734 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.734 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.734 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.675931+0000", 2024-04-19T02:03:26.734 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.015841133 2024-04-19T02:03:26.734 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.734 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.734 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.734 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.734 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.734 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpRead(3.2s0 32/26 ECSubRead(tid=75, to_read={3:407ecba7:::smithi05930959-22:head=0,90112,0}, subchunks={3:407ecba7:::smithi05930959-22:head=[0,1]}, attrs_to_read=))", 2024-04-19T02:03:26.734 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.660621+0000", 2024-04-19T02:03:26.735 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 10.026738747, 2024-04-19T02:03:26.735 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0067211509999999999, 2024-04-19T02:03:26.735 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.735 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.735 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:26.735 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.735 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.735 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.735 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.660621+0000", 2024-04-19T02:03:26.735 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.735 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.735 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.735 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.735 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.660621+0000", 2024-04-19T02:03:26.735 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.735 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.736 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.736 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.736 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.660621+0000", 2024-04-19T02:03:26.736 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.736 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.736 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.736 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.736 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.660621+0000", 2024-04-19T02:03:26.736 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.736 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.736 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.736 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.736 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.660667+0000", 2024-04-19T02:03:26.736 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.6139e-05 2024-04-19T02:03:26.736 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.737 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.737 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.737 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.660695+0000", 2024-04-19T02:03:26.737 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7093e-05 2024-04-19T02:03:26.737 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.737 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.737 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.737 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.664582+0000", 2024-04-19T02:03:26.737 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0038879299999999999 2024-04-19T02:03:26.737 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.737 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.737 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.737 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.667342+0000", 2024-04-19T02:03:26.737 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002759989 2024-04-19T02:03:26.738 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.738 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.738 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.738 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.738 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.738 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.2s0 32/26 ECSubReadReply(tid=75, attrs_read=0))", 2024-04-19T02:03:26.738 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.661256+0000", 2024-04-19T02:03:26.738 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 10.026104118999999, 2024-04-19T02:03:26.738 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0067562519999999999, 2024-04-19T02:03:26.738 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.738 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.738 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:26.738 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.738 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.739 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.739 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.661256+0000", 2024-04-19T02:03:26.739 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.739 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.739 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.739 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.739 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.661256+0000", 2024-04-19T02:03:26.739 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.739 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.739 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.739 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.739 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.661286+0000", 2024-04-19T02:03:26.739 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9921999999999999e-05 2024-04-19T02:03:26.740 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.740 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.740 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.740 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.661468+0000", 2024-04-19T02:03:26.740 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000181884 2024-04-19T02:03:26.740 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.740 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.740 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.740 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.661503+0000", 2024-04-19T02:03:26.740 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5685999999999997e-05 2024-04-19T02:03:26.740 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.740 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.740 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.740 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.661521+0000", 2024-04-19T02:03:26.740 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7728999999999999e-05 2024-04-19T02:03:26.741 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.741 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.741 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.741 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.667883+0000", 2024-04-19T02:03:26.741 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.006362075 2024-04-19T02:03:26.741 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.741 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.741 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.741 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.668012+0000", 2024-04-19T02:03:26.741 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000128956 2024-04-19T02:03:26.741 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.741 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.741 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.741 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.742 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.742 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.2s0 32/26 ECSubReadReply(tid=73, attrs_read=0))", 2024-04-19T02:03:26.742 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.662994+0000", 2024-04-19T02:03:26.742 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 10.024366516000001, 2024-04-19T02:03:26.742 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0078037799999999997, 2024-04-19T02:03:26.742 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.742 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.742 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:26.742 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.742 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.742 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.742 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.662994+0000", 2024-04-19T02:03:26.742 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.742 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.742 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.743 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.743 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.662994+0000", 2024-04-19T02:03:26.743 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.743 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.743 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.743 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.743 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.662994+0000", 2024-04-19T02:03:26.743 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.743 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.743 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.743 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.743 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.662994+0000", 2024-04-19T02:03:26.743 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.743 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.743 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.744 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.744 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.663041+0000", 2024-04-19T02:03:26.744 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.7849000000000002e-05 2024-04-19T02:03:26.744 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.744 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.744 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.744 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.663079+0000", 2024-04-19T02:03:26.744 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.7832e-05 2024-04-19T02:03:26.744 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.744 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.744 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.744 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.669809+0000", 2024-04-19T02:03:26.744 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0067298530000000001 2024-04-19T02:03:26.744 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.744 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.745 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.745 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.670797+0000", 2024-04-19T02:03:26.745 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00098824599999999992 2024-04-19T02:03:26.745 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.745 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.745 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.745 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.745 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.745 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.2s0 32/26 ECSubReadReply(tid=75, attrs_read=0))", 2024-04-19T02:03:26.745 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.663343+0000", 2024-04-19T02:03:26.745 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 10.024017247, 2024-04-19T02:03:26.745 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.011502955, 2024-04-19T02:03:26.745 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.746 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.746 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:26.746 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.746 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.746 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.746 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.663343+0000", 2024-04-19T02:03:26.746 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.746 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.746 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.746 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.746 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.663343+0000", 2024-04-19T02:03:26.746 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.746 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.746 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.746 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.747 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.663345+0000", 2024-04-19T02:03:26.747 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.153e-06 2024-04-19T02:03:26.747 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.747 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.747 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.747 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.663576+0000", 2024-04-19T02:03:26.747 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000231103 2024-04-19T02:03:26.747 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.747 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.747 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.747 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.663601+0000", 2024-04-19T02:03:26.747 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5113000000000002e-05 2024-04-19T02:03:26.747 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.747 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.747 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.748 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.663618+0000", 2024-04-19T02:03:26.748 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6362999999999999e-05 2024-04-19T02:03:26.748 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.748 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.748 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.748 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.674769+0000", 2024-04-19T02:03:26.748 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.011151978999999999 2024-04-19T02:03:26.748 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.748 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.748 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.748 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.674846+0000", 2024-04-19T02:03:26.748 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6243999999999997e-05 2024-04-19T02:03:26.748 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.748 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.748 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.748 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.749 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.749 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.2s0 32/26 ECSubReadReply(tid=74, attrs_read=0))", 2024-04-19T02:03:26.749 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.663949+0000", 2024-04-19T02:03:26.749 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 10.023410678999999, 2024-04-19T02:03:26.749 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.011325732999999999, 2024-04-19T02:03:26.749 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.749 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.749 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:26.749 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.749 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.749 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.749 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.663949+0000", 2024-04-19T02:03:26.749 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.749 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.749 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.750 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.750 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.663949+0000", 2024-04-19T02:03:26.750 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.750 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.750 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.750 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.750 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.663949+0000", 2024-04-19T02:03:26.750 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.750 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.750 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.750 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.750 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.663949+0000", 2024-04-19T02:03:26.750 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.750 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.750 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.751 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.751 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.664082+0000", 2024-04-19T02:03:26.751 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013250600000000001 2024-04-19T02:03:26.751 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.751 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.751 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.751 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.664122+0000", 2024-04-19T02:03:26.751 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0055999999999997e-05 2024-04-19T02:03:26.751 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.751 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.751 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.751 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.674882+0000", 2024-04-19T02:03:26.751 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010760525 2024-04-19T02:03:26.751 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.751 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.752 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.752 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.675275+0000", 2024-04-19T02:03:26.752 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00039264600000000002 2024-04-19T02:03:26.752 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.752 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.752 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.752 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.752 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.752 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.2s0 32/26 ECSubReadReply(tid=75, attrs_read=0))", 2024-04-19T02:03:26.752 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.667310+0000", 2024-04-19T02:03:26.752 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 10.020050475, 2024-04-19T02:03:26.752 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0086427229999999997, 2024-04-19T02:03:26.752 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.752 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.753 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:26.753 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.753 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.753 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.753 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.667310+0000", 2024-04-19T02:03:26.753 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.753 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.753 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.753 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.753 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.667310+0000", 2024-04-19T02:03:26.753 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.753 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.754 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.754 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.754 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.667310+0000", 2024-04-19T02:03:26.754 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.754 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.754 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.754 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.754 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.667310+0000", 2024-04-19T02:03:26.754 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.754 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.754 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.754 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.754 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.667358+0000", 2024-04-19T02:03:26.754 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.8041000000000003e-05 2024-04-19T02:03:26.755 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.755 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.755 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.755 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.667374+0000", 2024-04-19T02:03:26.755 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.658e-05 2024-04-19T02:03:26.755 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.755 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.755 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.755 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.675516+0000", 2024-04-19T02:03:26.755 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0081422189999999992 2024-04-19T02:03:26.755 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.755 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.755 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.755 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.675952+0000", 2024-04-19T02:03:26.755 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00043588299999999997 2024-04-19T02:03:26.756 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.756 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.756 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.756 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.756 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.756 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:349 3.2s0 3:43b7722e:::smithi05930959-10:head [copy-from ver 11] snapc 8=[8,6,5] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:26.756 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.746333+0000", 2024-04-19T02:03:26.756 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.941026651, 2024-04-19T02:03:26.756 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.034219922999999999, 2024-04-19T02:03:26.756 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.756 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.756 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:26.756 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:26.756 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:26.757 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:26.757 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 349 2024-04-19T02:03:26.757 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.757 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.757 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.757 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.757 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746333+0000", 2024-04-19T02:03:26.757 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.757 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.757 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.757 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.757 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746333+0000", 2024-04-19T02:03:26.757 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.757 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.757 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.757 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.758 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746337+0000", 2024-04-19T02:03:26.758 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4769999999999999e-06 2024-04-19T02:03:26.758 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.758 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.758 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.758 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746352+0000", 2024-04-19T02:03:26.758 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5277e-05 2024-04-19T02:03:26.758 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.758 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.758 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.758 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746387+0000", 2024-04-19T02:03:26.758 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4436e-05 2024-04-19T02:03:26.758 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.758 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.759 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.759 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746403+0000", 2024-04-19T02:03:26.759 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6475999999999999e-05 2024-04-19T02:03:26.759 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.759 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.759 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.759 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746490+0000", 2024-04-19T02:03:26.759 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7213000000000007e-05 2024-04-19T02:03:26.759 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.759 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.759 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "op must wait for map", 2024-04-19T02:03:26.759 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746497+0000", 2024-04-19T02:03:26.759 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.0520000000000004e-06 2024-04-19T02:03:26.759 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.759 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.759 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.760 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749865+0000", 2024-04-19T02:03:26.760 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0033676069999999999 2024-04-19T02:03:26.760 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.760 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.760 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:26.760 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749938+0000", 2024-04-19T02:03:26.760 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.3530999999999998e-05 2024-04-19T02:03:26.760 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.760 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.760 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:26.760 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.768855+0000", 2024-04-19T02:03:26.760 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.018916493 2024-04-19T02:03:26.760 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.760 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.760 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:26.761 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.771377+0000", 2024-04-19T02:03:26.761 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0025221520000000002 2024-04-19T02:03:26.761 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.761 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.761 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:26.761 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.780460+0000", 2024-04-19T02:03:26.761 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0090830389999999993 2024-04-19T02:03:26.761 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.761 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.761 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.761 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.780553+0000", 2024-04-19T02:03:26.761 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3170000000000004e-05 2024-04-19T02:03:26.761 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.761 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.761 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.762 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.762 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.762 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:350 3.2s0 3:43b7722e:::smithi05930959-10:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:26.762 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.746450+0000", 2024-04-19T02:03:26.762 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.9409099919999999, 2024-04-19T02:03:26.762 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.024575331999999998, 2024-04-19T02:03:26.762 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.762 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.762 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:26.762 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:26.762 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:26.762 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:26.762 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 350 2024-04-19T02:03:26.762 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.762 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.763 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.763 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.763 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746450+0000", 2024-04-19T02:03:26.763 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.763 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.763 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.763 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.763 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746450+0000", 2024-04-19T02:03:26.763 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.763 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.763 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.763 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.763 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746451+0000", 2024-04-19T02:03:26.763 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0400000000000005e-07 2024-04-19T02:03:26.763 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.763 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.764 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.764 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746456+0000", 2024-04-19T02:03:26.764 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5199999999999997e-06 2024-04-19T02:03:26.764 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.764 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.764 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.764 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746470+0000", 2024-04-19T02:03:26.764 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3935e-05 2024-04-19T02:03:26.764 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.764 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.764 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.764 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746481+0000", 2024-04-19T02:03:26.764 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0919e-05 2024-04-19T02:03:26.764 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.764 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.765 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.765 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746552+0000", 2024-04-19T02:03:26.765 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.0235999999999996e-05 2024-04-19T02:03:26.765 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.765 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.765 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:03:26.765 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746557+0000", 2024-04-19T02:03:26.765 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.6570000000000002e-06 2024-04-19T02:03:26.765 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.765 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.765 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.765 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750143+0000", 2024-04-19T02:03:26.765 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0035860549999999999 2024-04-19T02:03:26.765 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.765 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.765 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for blocked object", 2024-04-19T02:03:26.766 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750177+0000", 2024-04-19T02:03:26.766 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3796e-05 2024-04-19T02:03:26.766 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.766 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.766 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.766 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.770842+0000", 2024-04-19T02:03:26.766 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.020664657999999999 2024-04-19T02:03:26.766 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.766 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.766 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:26.766 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.770923+0000", 2024-04-19T02:03:26.766 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0869000000000001e-05 2024-04-19T02:03:26.766 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.766 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.766 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.767 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.771025+0000", 2024-04-19T02:03:26.767 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010278299999999999 2024-04-19T02:03:26.767 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.767 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.767 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.767 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.767 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.767 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:356 3.2s0 3:43ce42f3:::smithi05930959-27:head [read 0~1343488,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:26.767 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.772104+0000", 2024-04-19T02:03:26.767 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.91525642, 2024-04-19T02:03:26.767 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0071622250000000004, 2024-04-19T02:03:26.767 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.767 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.767 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:26.767 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:26.768 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:26.768 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:26.768 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 356 2024-04-19T02:03:26.768 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.768 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.768 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.768 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.768 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772104+0000", 2024-04-19T02:03:26.768 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.768 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.768 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.768 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.768 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772104+0000", 2024-04-19T02:03:26.768 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.769 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.769 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.769 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.769 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772106+0000", 2024-04-19T02:03:26.769 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0250000000000001e-06 2024-04-19T02:03:26.769 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.769 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.769 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.769 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772116+0000", 2024-04-19T02:03:26.769 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0603e-05 2024-04-19T02:03:26.769 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.769 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.769 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.769 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772133+0000", 2024-04-19T02:03:26.769 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7207999999999999e-05 2024-04-19T02:03:26.770 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.770 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.770 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.770 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772142+0000", 2024-04-19T02:03:26.770 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7660000000000004e-06 2024-04-19T02:03:26.770 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.770 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.770 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.770 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772236+0000", 2024-04-19T02:03:26.770 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.4042000000000005e-05 2024-04-19T02:03:26.770 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.770 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.770 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:26.770 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772348+0000", 2024-04-19T02:03:26.770 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000111591 2024-04-19T02:03:26.771 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.771 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.771 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.771 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.779266+0000", 2024-04-19T02:03:26.771 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0069179899999999997 2024-04-19T02:03:26.771 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.771 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.771 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.771 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.771 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.771 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:357 3.2s0 3:43ce42f3:::smithi05930959-27:head [read 0~1343488,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:26.771 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.772163+0000", 2024-04-19T02:03:26.771 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.915197037, 2024-04-19T02:03:26.771 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0081243819999999994, 2024-04-19T02:03:26.771 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.772 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.772 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:26.772 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:26.772 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:26.772 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:26.772 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 357 2024-04-19T02:03:26.772 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.772 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.772 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.772 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.772 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772163+0000", 2024-04-19T02:03:26.772 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.772 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.773 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.773 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.773 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772163+0000", 2024-04-19T02:03:26.773 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.773 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.773 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.773 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.773 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772164+0000", 2024-04-19T02:03:26.773 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.0800000000000004e-07 2024-04-19T02:03:26.773 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.773 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.773 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.773 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772168+0000", 2024-04-19T02:03:26.774 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4440000000000001e-06 2024-04-19T02:03:26.774 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.774 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.774 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.774 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772182+0000", 2024-04-19T02:03:26.774 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4218e-05 2024-04-19T02:03:26.774 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.774 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.774 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.774 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772190+0000", 2024-04-19T02:03:26.774 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.3950000000000003e-06 2024-04-19T02:03:26.774 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.774 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.774 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.774 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772768+0000", 2024-04-19T02:03:26.775 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00057860099999999999 2024-04-19T02:03:26.775 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.775 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.775 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:26.775 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772831+0000", 2024-04-19T02:03:26.775 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.2862999999999998e-05 2024-04-19T02:03:26.775 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.775 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.775 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.775 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.780287+0000", 2024-04-19T02:03:26.775 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0074561530000000001 2024-04-19T02:03:26.775 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.775 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.775 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.775 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.775 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.776 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:358 3.2s0 3:43ce42f3:::smithi05930959-27:head [read 0~1343488,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:26.776 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.772451+0000", 2024-04-19T02:03:26.776 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.914909226, 2024-04-19T02:03:26.776 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0096851239999999998, 2024-04-19T02:03:26.776 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:26.776 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:26.776 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:26.776 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:26.776 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:26.776 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:26.776 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 358 2024-04-19T02:03:26.776 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.776 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:26.776 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.777 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:26.777 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772451+0000", 2024-04-19T02:03:26.777 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.777 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.777 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.777 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:26.777 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772451+0000", 2024-04-19T02:03:26.777 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:26.777 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.777 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.777 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:26.777 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772452+0000", 2024-04-19T02:03:26.777 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3099999999999996e-07 2024-04-19T02:03:26.777 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.777 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.777 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:26.778 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772457+0000", 2024-04-19T02:03:26.778 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.6899999999999997e-06 2024-04-19T02:03:26.778 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.778 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.778 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:26.778 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772470+0000", 2024-04-19T02:03:26.778 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2336e-05 2024-04-19T02:03:26.778 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.778 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.778 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:26.778 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.772478+0000", 2024-04-19T02:03:26.778 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.8049999999999996e-06 2024-04-19T02:03:26.778 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.778 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.779 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:26.779 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.773118+0000", 2024-04-19T02:03:26.779 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00063943599999999995 2024-04-19T02:03:26.779 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.779 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.779 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:26.779 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.773202+0000", 2024-04-19T02:03:26.779 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3683999999999994e-05 2024-04-19T02:03:26.779 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:26.779 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:26.779 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:26.779 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.782136+0000", 2024-04-19T02:03:26.779 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0089343419999999996 2024-04-19T02:03:26.779 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.779 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.780 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.780 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:26.780 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:26.780 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:26.780 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T02:03:26.782 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-19T02:03:26.782 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.0 to 1 (10000) 2024-04-19T02:03:26.783 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is unset 2024-04-19T02:03:26.784 INFO:tasks.rados.rados.0.smithi059.stdout:139: done (0 left) 2024-04-19T02:03:26.784 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 38 current snap is 8 2024-04-19T02:03:26.784 INFO:tasks.rados.rados.0.smithi059.stdout:140: seq_num 83 ranges {0=638976,638976=622592,1261568=147456} 2024-04-19T02:03:26.784 INFO:tasks.rados.rados.0.smithi059.stdout:140: writing smithi05930959-38 from 0 to 638976 tid 1 2024-04-19T02:03:26.784 INFO:tasks.rados.rados.0.smithi059.stdout:140: writing smithi05930959-38 from 638976 to 1261568 tid 2 2024-04-19T02:03:26.784 INFO:tasks.rados.rados.0.smithi059.stdout:140: writing smithi05930959-38 from 1261568 to 1409024 tid 3 2024-04-19T02:03:26.784 INFO:tasks.rados.rados.0.smithi059.stdout:141: read oid 23 snap -1 2024-04-19T02:03:26.784 INFO:tasks.rados.rados.0.smithi059.stdout:141: expect (ObjNum 62 snap 2 seq_num 62) 2024-04-19T02:03:26.785 INFO:tasks.rados.rados.0.smithi059.stdout:142: read oid 20 snap -1 2024-04-19T02:03:26.785 INFO:tasks.rados.rados.0.smithi059.stdout:142: expect (ObjNum 76 snap 6 seq_num 76) 2024-04-19T02:03:26.785 INFO:tasks.rados.rados.0.smithi059.stdout:143: rollback oid 44 current snap is 8 2024-04-19T02:03:26.785 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 44 to 6 2024-04-19T02:03:26.785 INFO:tasks.rados.rados.0.smithi059.stdout:144: read oid 42 snap -1 2024-04-19T02:03:26.785 INFO:tasks.rados.rados.0.smithi059.stdout:144: expect (ObjNum 35 snap 0 seq_num 35) 2024-04-19T02:03:26.785 INFO:tasks.rados.rados.0.smithi059.stdout:145: snap_remove snap 7 2024-04-19T02:03:26.786 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-19T02:03:26.787 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:26.783+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:26.882 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:26.882+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:26.917 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:26.917 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:26.918 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 0 2024-04-19T02:03:26.918 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:26.931 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-19T02:03:26.968 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-04-19T02:03:26.968 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T02:03:26.985 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-04-19T02:03:26.985 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T02:03:27.001 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-04-19T02:03:27.001 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T02:03:27.018 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:27.018+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:27.019 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-04-19T02:03:27.019 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T02:03:27.037 INFO:tasks.rados.rados.0.smithi059.stdout:145: done (5 left) 2024-04-19T02:03:27.037 INFO:tasks.rados.rados.0.smithi059.stdout:146: setattr oid 48 current snap is 8 2024-04-19T02:03:27.037 INFO:tasks.rados.rados.0.smithi059.stdout:141: expect (ObjNum 62 snap 2 seq_num 62) 2024-04-19T02:03:27.039 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-04-19T02:03:27.039 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T02:03:27.058 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-04-19T02:03:27.058 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T02:03:27.076 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-04-19T02:03:27.076 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T02:03:27.082 INFO:tasks.rados.rados.0.smithi059.stdout:140: finishing write tid 1 to smithi05930959-38 2024-04-19T02:03:27.082 INFO:tasks.rados.rados.0.smithi059.stdout:140: finishing write tid 2 to smithi05930959-38 2024-04-19T02:03:27.082 INFO:tasks.rados.rados.0.smithi059.stdout:143: finishing rollback tid 0 to smithi05930959-44 2024-04-19T02:03:27.082 INFO:tasks.rados.rados.0.smithi059.stdout:143: finishing rollback tid 1 to smithi05930959-44 2024-04-19T02:03:27.082 INFO:tasks.rados.rados.0.smithi059.stdout:143: finishing rollback tid 2 to smithi05930959-44 2024-04-19T02:03:27.082 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 44 v 36 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-19T02:03:27.082 INFO:tasks.rados.rados.0.smithi059.stdout:140: finishing write tid 3 to smithi05930959-38 2024-04-19T02:03:27.082 INFO:tasks.rados.rados.0.smithi059.stdout:140: finishing write tid 4 to smithi05930959-38 2024-04-19T02:03:27.082 INFO:tasks.rados.rados.0.smithi059.stdout:144: expect (ObjNum 35 snap 0 seq_num 35) 2024-04-19T02:03:27.083 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:27.083+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:27.099 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-04-19T02:03:27.100 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T02:03:27.101 INFO:tasks.rados.rados.0.smithi059.stdout:142: expect (ObjNum 76 snap 6 seq_num 76) 2024-04-19T02:03:27.106 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:27.106 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:27.121 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-19T02:03:27.124 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-04-19T02:03:27.125 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T02:03:27.146 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-04-19T02:03:27.146 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T02:03:27.154 INFO:tasks.rados.rados.0.smithi059.stdout:140: finishing write tid 5 to smithi05930959-38 2024-04-19T02:03:27.154 INFO:tasks.rados.rados.0.smithi059.stdout:140: oid 38 updating version 0 to 18 2024-04-19T02:03:27.154 INFO:tasks.rados.rados.0.smithi059.stdout:140: oid 38 version 18 is already newer than 15 2024-04-19T02:03:27.154 INFO:tasks.rados.rados.0.smithi059.stdout:140: oid 38 version 18 is already newer than 17 2024-04-19T02:03:27.154 INFO:tasks.rados.rados.0.smithi059.stdout:140: oid 38 version 18 is already newer than 16 2024-04-19T02:03:27.154 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 38 v 18 (ObjNum 83 snap 8 seq_num 83) dirty exists 2024-04-19T02:03:27.154 INFO:tasks.rados.rados.0.smithi059.stdout:140: left oid 38 (ObjNum 83 snap 8 seq_num 83) 2024-04-19T02:03:27.154 INFO:tasks.rados.rados.0.smithi059.stdout:140: done (5 left) 2024-04-19T02:03:27.154 INFO:tasks.rados.rados.0.smithi059.stdout:141: done (4 left) 2024-04-19T02:03:27.154 INFO:tasks.rados.rados.0.smithi059.stdout:142: done (3 left) 2024-04-19T02:03:27.154 INFO:tasks.rados.rados.0.smithi059.stdout:143: done (2 left) 2024-04-19T02:03:27.155 INFO:tasks.rados.rados.0.smithi059.stdout:144: done (1 left) 2024-04-19T02:03:27.155 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 16 current snap is 8 2024-04-19T02:03:27.155 INFO:tasks.rados.rados.0.smithi059.stdout:147: seq_num 84 ranges {262144=655360,917504=638976,1556480=131072} 2024-04-19T02:03:27.158 INFO:tasks.rados.rados.0.smithi059.stdout:147: writing smithi05930959-16 from 262144 to 917504 tid 1 2024-04-19T02:03:27.162 INFO:tasks.rados.rados.0.smithi059.stdout:147: writing smithi05930959-16 from 917504 to 1556480 tid 2 2024-04-19T02:03:27.163 INFO:tasks.rados.rados.0.smithi059.stdout:147: writing smithi05930959-16 from 1556480 to 1687552 tid 3 2024-04-19T02:03:27.163 INFO:tasks.rados.rados.0.smithi059.stdout:148: read oid 50 snap -1 2024-04-19T02:03:27.163 INFO:tasks.rados.rados.0.smithi059.stdout:148: expect (ObjNum 29 snap 0 seq_num 29) 2024-04-19T02:03:27.163 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 24 current snap is 8 2024-04-19T02:03:27.164 INFO:tasks.rados.rados.0.smithi059.stdout:149: seq_num 85 ranges {720896=671744,1392640=770048} 2024-04-19T02:03:27.170 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-04-19T02:03:27.171 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T02:03:27.171 INFO:tasks.rados.rados.0.smithi059.stdout:149: writing smithi05930959-24 from 720896 to 1392640 tid 1 2024-04-19T02:03:27.178 INFO:tasks.rados.rados.0.smithi059.stdout:149: writing smithi05930959-24 from 1392640 to 2162688 tid 2 2024-04-19T02:03:27.178 INFO:tasks.rados.rados.0.smithi059.stdout:150: snap_create 2024-04-19T02:03:27.179 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 48 v 36 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-19T02:03:27.179 INFO:tasks.rados.rados.0.smithi059.stdout:147: finishing write tid 1 to smithi05930959-16 2024-04-19T02:03:27.179 INFO:tasks.rados.rados.0.smithi059.stdout:147: finishing write tid 2 to smithi05930959-16 2024-04-19T02:03:27.180 INFO:tasks.rados.rados.0.smithi059.stdout:148: expect (ObjNum 29 snap 0 seq_num 29) 2024-04-19T02:03:27.202 INFO:tasks.rados.rados.0.smithi059.stdout:147: finishing write tid 3 to smithi05930959-16 2024-04-19T02:03:27.202 INFO:tasks.rados.rados.0.smithi059.stdout:149: finishing write tid 1 to smithi05930959-24 2024-04-19T02:03:27.202 INFO:tasks.rados.rados.0.smithi059.stdout:147: finishing write tid 4 to smithi05930959-16 2024-04-19T02:03:27.202 INFO:tasks.rados.rados.0.smithi059.stdout:149: finishing write tid 2 to smithi05930959-24 2024-04-19T02:03:27.202 INFO:tasks.rados.rados.0.smithi059.stdout:147: finishing write tid 5 to smithi05930959-16 2024-04-19T02:03:27.202 INFO:tasks.rados.rados.0.smithi059.stdout:147: oid 16 updating version 0 to 23 2024-04-19T02:03:27.202 INFO:tasks.rados.rados.0.smithi059.stdout:147: oid 16 version 23 is already newer than 20 2024-04-19T02:03:27.202 INFO:tasks.rados.rados.0.smithi059.stdout:147: oid 16 version 23 is already newer than 21 2024-04-19T02:03:27.202 INFO:tasks.rados.rados.0.smithi059.stdout:147: oid 16 version 23 is already newer than 22 2024-04-19T02:03:27.202 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 16 v 23 (ObjNum 84 snap 8 seq_num 84) dirty exists 2024-04-19T02:03:27.203 INFO:tasks.rados.rados.0.smithi059.stdout:147: left oid 16 (ObjNum 84 snap 8 seq_num 84) 2024-04-19T02:03:27.203 INFO:tasks.rados.rados.0.smithi059.stdout:149: finishing write tid 3 to smithi05930959-24 2024-04-19T02:03:27.203 INFO:tasks.rados.rados.0.smithi059.stdout:149: finishing write tid 4 to smithi05930959-24 2024-04-19T02:03:27.203 INFO:tasks.rados.rados.0.smithi059.stdout:149: oid 24 updating version 0 to 8 2024-04-19T02:03:27.203 INFO:tasks.rados.rados.0.smithi059.stdout:149: oid 24 updating version 8 to 10 2024-04-19T02:03:27.203 INFO:tasks.rados.rados.0.smithi059.stdout:149: oid 24 version 10 is already newer than 9 2024-04-19T02:03:27.203 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 24 v 10 (ObjNum 85 snap 8 seq_num 85) dirty exists 2024-04-19T02:03:27.203 INFO:tasks.rados.rados.0.smithi059.stdout:149: left oid 24 (ObjNum 85 snap 8 seq_num 85) 2024-04-19T02:03:27.204 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-04-19T02:03:27.204 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T02:03:27.210 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:27.210+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:27.285 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:27.286+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:27.290 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:27.290 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:27.304 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-19T02:03:27.390 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:27.390+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:27.469 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:27.469 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:27.482 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-19T02:03:27.526 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:27.526+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:27.588 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:27.587+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:27.646 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:27.646 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:27.660 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-19T02:03:27.690 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:27.687+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:27.782 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:27.782 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:27.790 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:27.791+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:27.796 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-19T02:03:27.891 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:27.891+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:27.921 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:27.921 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:27.935 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-19T02:03:27.992 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:27.994+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:28.038 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.1 to 1 (10000) 2024-04-19T02:03:28.045 INFO:tasks.rados.rados.0.smithi059.stdout:146: done (4 left) 2024-04-19T02:03:28.046 INFO:tasks.rados.rados.0.smithi059.stdout:147: done (3 left) 2024-04-19T02:03:28.046 INFO:tasks.rados.rados.0.smithi059.stdout:148: done (2 left) 2024-04-19T02:03:28.046 INFO:tasks.rados.rados.0.smithi059.stdout:149: done (1 left) 2024-04-19T02:03:28.046 INFO:tasks.rados.rados.0.smithi059.stdout:150: done (0 left) 2024-04-19T02:03:28.046 INFO:tasks.rados.rados.0.smithi059.stdout:151: copy_from oid 4 from oid 34 current snap is 9 2024-04-19T02:03:28.046 INFO:tasks.rados.rados.0.smithi059.stdout:152: snap_remove snap 4 2024-04-19T02:03:28.062 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:28.062 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:28.067 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-19T02:03:28.078 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-19T02:03:28.110 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:28.111+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:28.196 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:28.194+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:28.198 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:28.198 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:28.213 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-19T02:03:28.296 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:28.295+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:28.385 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:28.385 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:28.396 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:28.394+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:28.399 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-19T02:03:28.497 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:28.498+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:28.566 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:28.567 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:28.581 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-19T02:03:28.598 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:28.595+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:28.699 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:28.698+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:28.703 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:28.704 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:28.718 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-19T02:03:28.788 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-19T02:03:28.802 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:28.802+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:28.850 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:28.850 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:28.863 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-19T02:03:28.900 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:28.899+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:28.984 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:28.985 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:28.998 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-19T02:03:29.001 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:29.002+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:29.046 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.2 to 1 (10000) 2024-04-19T02:03:29.062 INFO:tasks.rados.rados.0.smithi059.stdout:152: done (1 left) 2024-04-19T02:03:29.062 INFO:tasks.rados.rados.0.smithi059.stdout:153: read oid 39 snap 8 2024-04-19T02:03:29.062 INFO:tasks.rados.rados.0.smithi059.stdout:153: expect (ObjNum 78 snap 6 seq_num 78) 2024-04-19T02:03:29.062 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 41 current snap is 9 2024-04-19T02:03:29.062 INFO:tasks.rados.rados.0.smithi059.stdout:154: seq_num 86 ranges {1835008=688128,2523136=540672,3063808=229376} 2024-04-19T02:03:29.092 INFO:tasks.rados.rados.0.smithi059.stdout:154: writing smithi05930959-41 from 1835008 to 2523136 tid 1 2024-04-19T02:03:29.101 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:29.098+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:29.106 INFO:tasks.rados.rados.0.smithi059.stdout:154: writing smithi05930959-41 from 2523136 to 3063808 tid 2 2024-04-19T02:03:29.107 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-19T02:03:29.108 INFO:tasks.rados.rados.0.smithi059.stdout:154: writing smithi05930959-41 from 3063808 to 3293184 tid 3 2024-04-19T02:03:29.109 INFO:tasks.rados.rados.0.smithi059.stdout:155: read oid 38 snap -1 2024-04-19T02:03:29.109 INFO:tasks.rados.rados.0.smithi059.stdout:155: expect (ObjNum 83 snap 8 seq_num 83) 2024-04-19T02:03:29.109 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 40 current snap is 9 2024-04-19T02:03:29.109 INFO:tasks.rados.rados.0.smithi059.stdout:151: finishing copy_from racing read to smithi05930959-4 2024-04-19T02:03:29.109 INFO:tasks.rados.rados.0.smithi059.stdout:151: finishing copy_from to smithi05930959-4 2024-04-19T02:03:29.109 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 4 v 23 (ObjNum 82 snap 7 seq_num 82) dirty exists 2024-04-19T02:03:29.109 INFO:tasks.rados.rados.0.smithi059.stdout:156: seq_num 87 ranges {1687552=704512,2392064=688128,3080192=81920} 2024-04-19T02:03:29.123 INFO:tasks.rados.rados.0.smithi059.stdout:156: writing smithi05930959-40 from 1687552 to 2392064 tid 1 2024-04-19T02:03:29.127 INFO:tasks.rados.rados.0.smithi059.stdout:156: writing smithi05930959-40 from 2392064 to 3080192 tid 2 2024-04-19T02:03:29.128 INFO:tasks.rados.rados.0.smithi059.stdout:156: writing smithi05930959-40 from 3080192 to 3162112 tid 3 2024-04-19T02:03:29.128 INFO:tasks.rados.rados.0.smithi059.stdout:151: done (4 left) 2024-04-19T02:03:29.128 INFO:tasks.rados.rados.0.smithi059.stdout:157: snap_remove snap 6 2024-04-19T02:03:29.137 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:29.137 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:29.153 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-19T02:03:29.202 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:29.202+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:29.220 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T02:03:29.271 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:29.271 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:29.285 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-19T02:03:29.302 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:29.302+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:29.378 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-04-19T02:03:29.378 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T02:03:29.394 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-04-19T02:03:29.395 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T02:03:29.403 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:29.403 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:29.403 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:29.402+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:29.411 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-04-19T02:03:29.411 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T02:03:29.417 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-19T02:03:29.428 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-04-19T02:03:29.428 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T02:03:29.449 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-04-19T02:03:29.449 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T02:03:29.470 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-04-19T02:03:29.470 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T02:03:29.491 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-04-19T02:03:29.491 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T02:03:29.509 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:29.510+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:29.512 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-04-19T02:03:29.512 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T02:03:29.532 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-04-19T02:03:29.532 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T02:03:29.554 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-04-19T02:03:29.554 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T02:03:29.577 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-04-19T02:03:29.577 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T02:03:29.589 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:29.589 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-04-19T02:03:29.589 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-04-19T02:03:29.589 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-04-19T02:03:29.589 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.589 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s7 32/26 ECSubWrite(tid=72, reqid=client.4509.0:236, at_version=32'29, trim_to=0'0, roll_forward_to=31'28))", 2024-04-19T02:03:29.589 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:16.655346+0000", 2024-04-19T02:03:29.589 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 12.934751974999999, 2024-04-19T02:03:29.589 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011339666, 2024-04-19T02:03:29.590 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:29.590 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:29.590 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:29.590 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:29.590 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.590 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:29.590 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.655346+0000", 2024-04-19T02:03:29.590 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.590 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.590 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.590 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:29.590 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.655346+0000", 2024-04-19T02:03:29.590 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.591 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.591 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.591 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:29.591 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.655346+0000", 2024-04-19T02:03:29.591 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4999999999999998e-07 2024-04-19T02:03:29.591 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.591 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.591 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:29.591 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.655549+0000", 2024-04-19T02:03:29.591 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020310900000000001 2024-04-19T02:03:29.591 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.591 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.591 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:29.591 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.655578+0000", 2024-04-19T02:03:29.591 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8595999999999999e-05 2024-04-19T02:03:29.592 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.592 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.592 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:29.592 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.655602+0000", 2024-04-19T02:03:29.592 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3896999999999998e-05 2024-04-19T02:03:29.592 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.592 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.592 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:29.592 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.655777+0000", 2024-04-19T02:03:29.592 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000174935 2024-04-19T02:03:29.592 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.592 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.592 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:29.592 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.655803+0000", 2024-04-19T02:03:29.592 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6313e-05 2024-04-19T02:03:29.592 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.593 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.593 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:29.593 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.666585+0000", 2024-04-19T02:03:29.593 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010781677 2024-04-19T02:03:29.593 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.593 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.593 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:29.593 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.666685+0000", 2024-04-19T02:03:29.593 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000100789 2024-04-19T02:03:29.593 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.593 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:29.593 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.593 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.593 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.593 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s7 40/26 ECSubWrite(tid=127, reqid=client.4509.0:349, at_version=40'61, trim_to=0'0, roll_forward_to=39'59))", 2024-04-19T02:03:29.594 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:24.769504+0000", 2024-04-19T02:03:29.594 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4.8205939510000002, 2024-04-19T02:03:29.594 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0069576890000000004, 2024-04-19T02:03:29.594 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:29.594 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:29.594 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:29.594 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:29.594 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.594 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:29.594 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.769504+0000", 2024-04-19T02:03:29.594 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.594 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.594 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.594 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:29.595 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.769504+0000", 2024-04-19T02:03:29.595 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.595 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.595 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.595 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:29.595 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.769504+0000", 2024-04-19T02:03:29.595 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5700000000000002e-07 2024-04-19T02:03:29.595 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.595 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.595 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:29.595 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.770800+0000", 2024-04-19T02:03:29.595 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001295521 2024-04-19T02:03:29.595 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.595 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.596 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:29.596 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.770831+0000", 2024-04-19T02:03:29.596 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1483999999999999e-05 2024-04-19T02:03:29.596 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.596 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.596 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:29.596 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.770854+0000", 2024-04-19T02:03:29.596 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2175e-05 2024-04-19T02:03:29.596 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.596 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.596 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:29.596 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.771056+0000", 2024-04-19T02:03:29.596 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020260300000000001 2024-04-19T02:03:29.596 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.597 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.597 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:29.597 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.771075+0000", 2024-04-19T02:03:29.597 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8766e-05 2024-04-19T02:03:29.597 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.597 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.597 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:29.597 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.776356+0000", 2024-04-19T02:03:29.597 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0052808040000000001 2024-04-19T02:03:29.597 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.597 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.597 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:29.597 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.776462+0000", 2024-04-19T02:03:29.597 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000105779 2024-04-19T02:03:29.597 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.598 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:29.598 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.598 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.598 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.598 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:359 3.cs0 3:3da93e5c:::smithi05930959-42:head [copy-from ver 6] snapc 8=[8,6,5] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:29.598 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:24.774030+0000", 2024-04-19T02:03:29.598 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4.8160677329999997, 2024-04-19T02:03:29.598 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022176986999999999, 2024-04-19T02:03:29.598 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:29.598 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:29.598 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:29.598 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:29.598 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:29.598 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:29.599 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 359 2024-04-19T02:03:29.599 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.599 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:29.599 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.599 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:29.599 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774030+0000", 2024-04-19T02:03:29.599 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.599 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.599 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.599 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:29.599 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774030+0000", 2024-04-19T02:03:29.599 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.599 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.599 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.599 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:29.599 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774033+0000", 2024-04-19T02:03:29.600 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3859999999999999e-06 2024-04-19T02:03:29.600 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.600 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.600 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:29.600 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774051+0000", 2024-04-19T02:03:29.600 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8031999999999999e-05 2024-04-19T02:03:29.600 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.600 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.600 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:29.600 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774086+0000", 2024-04-19T02:03:29.600 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4573999999999998e-05 2024-04-19T02:03:29.600 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.600 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.600 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:29.600 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774108+0000", 2024-04-19T02:03:29.601 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1931000000000001e-05 2024-04-19T02:03:29.601 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.601 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.601 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:29.601 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774260+0000", 2024-04-19T02:03:29.601 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015161499999999999 2024-04-19T02:03:29.601 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.601 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.601 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:29.601 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774532+0000", 2024-04-19T02:03:29.601 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00027218700000000001 2024-04-19T02:03:29.601 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.601 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.601 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:29.602 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.791272+0000", 2024-04-19T02:03:29.602 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016740655 2024-04-19T02:03:29.602 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.602 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.602 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:29.602 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.794135+0000", 2024-04-19T02:03:29.602 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0028628170000000001 2024-04-19T02:03:29.602 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.602 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.602 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:29.602 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.796005+0000", 2024-04-19T02:03:29.602 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0018697 2024-04-19T02:03:29.602 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.602 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.603 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:29.603 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.796207+0000", 2024-04-19T02:03:29.603 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020209000000000001 2024-04-19T02:03:29.603 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.603 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:29.603 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.603 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.603 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.603 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:360 3.cs0 3:3da93e5c:::smithi05930959-42:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:29.603 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:24.774163+0000", 2024-04-19T02:03:29.603 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4.8159347610000003, 2024-04-19T02:03:29.603 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019807084999999999, 2024-04-19T02:03:29.603 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:29.603 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:29.603 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:29.603 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:29.604 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:29.604 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:29.604 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 360 2024-04-19T02:03:29.604 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.604 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:29.604 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.604 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:29.604 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774163+0000", 2024-04-19T02:03:29.604 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.604 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.604 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.604 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:29.604 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774163+0000", 2024-04-19T02:03:29.604 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.604 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.604 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.605 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:29.605 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774165+0000", 2024-04-19T02:03:29.605 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.513e-06 2024-04-19T02:03:29.605 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.605 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.605 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:29.605 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774173+0000", 2024-04-19T02:03:29.605 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.32e-06 2024-04-19T02:03:29.605 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.605 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.605 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:29.605 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774196+0000", 2024-04-19T02:03:29.605 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3493999999999999e-05 2024-04-19T02:03:29.605 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.605 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.605 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:29.606 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774213+0000", 2024-04-19T02:03:29.606 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6835999999999999e-05 2024-04-19T02:03:29.606 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.606 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.606 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:29.606 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774977+0000", 2024-04-19T02:03:29.606 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00076389999999999997 2024-04-19T02:03:29.606 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.606 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.606 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for blocked object", 2024-04-19T02:03:29.606 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.775221+0000", 2024-04-19T02:03:29.606 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00024397399999999999 2024-04-19T02:03:29.606 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.607 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.607 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:29.607 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.793417+0000", 2024-04-19T02:03:29.607 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018195896 2024-04-19T02:03:29.607 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.607 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.607 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:29.607 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.793748+0000", 2024-04-19T02:03:29.607 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000330716 2024-04-19T02:03:29.607 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.607 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.607 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:29.607 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.793970+0000", 2024-04-19T02:03:29.607 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022243600000000001 2024-04-19T02:03:29.607 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.608 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:29.608 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.608 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.608 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.608 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.ds6 42/26 ECSubWrite(tid=163, reqid=osd.8.0:163, at_version=42'14, trim_to=0'0, roll_forward_to=39'12))", 2024-04-19T02:03:29.608 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.743451+0000", 2024-04-19T02:03:29.608 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 2.8466472110000001, 2024-04-19T02:03:29.608 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022917335, 2024-04-19T02:03:29.608 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:29.608 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:29.608 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:29.608 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:29.608 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.608 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:29.608 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.743451+0000", 2024-04-19T02:03:29.609 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.609 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.609 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.609 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:29.609 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.743451+0000", 2024-04-19T02:03:29.609 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.609 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.609 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.609 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:29.609 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.743452+0000", 2024-04-19T02:03:29.609 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.33e-07 2024-04-19T02:03:29.609 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.609 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.610 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:29.610 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.743513+0000", 2024-04-19T02:03:29.610 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1404999999999999e-05 2024-04-19T02:03:29.610 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.610 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.610 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:29.610 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.743548+0000", 2024-04-19T02:03:29.610 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5187e-05 2024-04-19T02:03:29.610 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.610 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.610 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:29.610 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.743573+0000", 2024-04-19T02:03:29.610 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4618e-05 2024-04-19T02:03:29.610 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.611 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.611 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:29.611 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.743799+0000", 2024-04-19T02:03:29.611 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000226553 2024-04-19T02:03:29.611 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.611 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.611 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:29.611 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.743825+0000", 2024-04-19T02:03:29.611 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6101e-05 2024-04-19T02:03:29.611 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.611 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.611 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:29.611 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.766270+0000", 2024-04-19T02:03:29.611 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022444664999999999 2024-04-19T02:03:29.612 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.612 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.612 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:29.612 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.766368+0000", 2024-04-19T02:03:29.612 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7873000000000003e-05 2024-04-19T02:03:29.612 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.612 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:29.612 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.612 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.612 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.612 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s7 42/26 ECSubWrite(tid=132, reqid=osd.3.0:132, at_version=42'63, trim_to=0'0, roll_forward_to=40'61))", 2024-04-19T02:03:29.612 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.747459+0000", 2024-04-19T02:03:29.612 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 2.8426388870000001, 2024-04-19T02:03:29.612 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.023151675, 2024-04-19T02:03:29.613 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:29.613 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:29.613 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:29.613 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:29.613 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.613 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:29.613 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747459+0000", 2024-04-19T02:03:29.613 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.613 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.613 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.613 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:29.613 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747459+0000", 2024-04-19T02:03:29.613 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.613 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.613 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.613 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:29.613 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747460+0000", 2024-04-19T02:03:29.614 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7000000000000004e-07 2024-04-19T02:03:29.614 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.614 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.614 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:29.614 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747515+0000", 2024-04-19T02:03:29.614 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5248999999999999e-05 2024-04-19T02:03:29.614 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.614 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.614 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:29.614 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747546+0000", 2024-04-19T02:03:29.614 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1244000000000001e-05 2024-04-19T02:03:29.614 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.614 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.614 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:29.614 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747569+0000", 2024-04-19T02:03:29.615 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2835000000000001e-05 2024-04-19T02:03:29.615 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.615 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.615 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:29.615 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747776+0000", 2024-04-19T02:03:29.615 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020755099999999999 2024-04-19T02:03:29.615 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.615 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.615 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:29.615 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747802+0000", 2024-04-19T02:03:29.615 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5694000000000002e-05 2024-04-19T02:03:29.615 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.615 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.615 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:29.615 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.770557+0000", 2024-04-19T02:03:29.616 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022754936 2024-04-19T02:03:29.616 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.616 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.616 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:29.616 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.770611+0000", 2024-04-19T02:03:29.616 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3495999999999997e-05 2024-04-19T02:03:29.616 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.616 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:29.616 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.616 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.616 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.616 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.5s3 42/26 ECSubWrite(tid=51, reqid=osd.10.0:51, at_version=42'27, trim_to=0'0, roll_forward_to=39'25))", 2024-04-19T02:03:29.616 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.750076+0000", 2024-04-19T02:03:29.617 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 2.8400215370000002, 2024-04-19T02:03:29.617 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020350849000000001, 2024-04-19T02:03:29.617 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:29.617 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:29.617 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:29.617 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:29.617 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.617 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:29.617 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750076+0000", 2024-04-19T02:03:29.617 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.617 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.617 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.617 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:29.617 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750076+0000", 2024-04-19T02:03:29.618 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.618 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.618 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.618 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:29.618 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750077+0000", 2024-04-19T02:03:29.618 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7000000000000004e-07 2024-04-19T02:03:29.618 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.618 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.618 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:29.618 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750133+0000", 2024-04-19T02:03:29.618 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6119000000000003e-05 2024-04-19T02:03:29.618 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.618 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.618 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:29.619 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750163+0000", 2024-04-19T02:03:29.619 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0328000000000002e-05 2024-04-19T02:03:29.619 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.619 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.619 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:29.619 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750220+0000", 2024-04-19T02:03:29.619 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6325e-05 2024-04-19T02:03:29.619 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.619 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.619 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:29.619 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750376+0000", 2024-04-19T02:03:29.619 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015590100000000001 2024-04-19T02:03:29.619 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.619 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.619 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:29.620 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750401+0000", 2024-04-19T02:03:29.620 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5635000000000001e-05 2024-04-19T02:03:29.620 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.620 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.620 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:29.620 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.770323+0000", 2024-04-19T02:03:29.620 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019921464999999999 2024-04-19T02:03:29.620 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.620 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.620 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:29.620 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.770427+0000", 2024-04-19T02:03:29.620 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000104306 2024-04-19T02:03:29.620 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.621 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:29.621 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.621 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.621 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.621 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.as3 42/26 ECSubWrite(tid=82, reqid=osd.11.0:82, at_version=42'34, trim_to=0'0, roll_forward_to=39'32))", 2024-04-19T02:03:29.621 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.750214+0000", 2024-04-19T02:03:29.621 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 2.8398835670000002, 2024-04-19T02:03:29.621 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019187731999999999, 2024-04-19T02:03:29.621 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:29.621 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:29.621 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:29.621 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:29.621 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.621 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:29.622 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750214+0000", 2024-04-19T02:03:29.622 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.622 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.622 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.622 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:29.622 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750214+0000", 2024-04-19T02:03:29.622 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.622 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.622 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.622 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:29.622 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750215+0000", 2024-04-19T02:03:29.622 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4000000000000001e-07 2024-04-19T02:03:29.622 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.622 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.623 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:29.623 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750268+0000", 2024-04-19T02:03:29.623 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3389e-05 2024-04-19T02:03:29.623 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.623 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.623 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:29.623 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750299+0000", 2024-04-19T02:03:29.623 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0914000000000003e-05 2024-04-19T02:03:29.623 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.623 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.623 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:29.623 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750322+0000", 2024-04-19T02:03:29.623 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2575e-05 2024-04-19T02:03:29.623 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.624 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.624 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:29.624 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750578+0000", 2024-04-19T02:03:29.624 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00025623300000000002 2024-04-19T02:03:29.624 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.624 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.624 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:29.624 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750606+0000", 2024-04-19T02:03:29.624 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7568000000000001e-05 2024-04-19T02:03:29.624 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.624 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.624 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:29.624 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.769319+0000", 2024-04-19T02:03:29.625 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018712909 2024-04-19T02:03:29.625 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.625 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.625 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:29.625 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.769402+0000", 2024-04-19T02:03:29.625 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3404000000000003e-05 2024-04-19T02:03:29.625 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.625 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:29.625 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.625 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.625 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.625 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s7 42/26 ECSubWrite(tid=134, reqid=osd.3.0:134, at_version=42'65, trim_to=0'0, roll_forward_to=40'61))", 2024-04-19T02:03:29.625 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.750993+0000", 2024-04-19T02:03:29.626 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 2.8391043950000001, 2024-04-19T02:03:29.626 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021026969, 2024-04-19T02:03:29.626 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:29.626 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:29.626 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:29.626 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:29.626 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.626 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:29.626 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750993+0000", 2024-04-19T02:03:29.626 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.626 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.626 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.626 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:29.626 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750993+0000", 2024-04-19T02:03:29.627 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.627 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.627 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.627 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:29.627 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750995+0000", 2024-04-19T02:03:29.627 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.883e-06 2024-04-19T02:03:29.627 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.627 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.627 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:29.627 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751070+0000", 2024-04-19T02:03:29.627 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4981000000000006e-05 2024-04-19T02:03:29.627 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.627 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.627 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:29.628 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751100+0000", 2024-04-19T02:03:29.628 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9944999999999999e-05 2024-04-19T02:03:29.628 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.628 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.628 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:29.628 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751122+0000", 2024-04-19T02:03:29.628 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1810999999999998e-05 2024-04-19T02:03:29.628 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.628 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.628 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:29.628 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.762958+0000", 2024-04-19T02:03:29.628 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011836187 2024-04-19T02:03:29.628 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.628 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.628 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:29.629 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.762984+0000", 2024-04-19T02:03:29.629 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5420999999999999e-05 2024-04-19T02:03:29.629 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.629 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.629 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:29.629 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.771936+0000", 2024-04-19T02:03:29.629 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0089524319999999998 2024-04-19T02:03:29.629 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.629 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.629 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:29.629 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.772020+0000", 2024-04-19T02:03:29.629 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4308999999999995e-05 2024-04-19T02:03:29.629 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.629 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:29.629 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.630 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.630 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.630 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.8s5 42/26 ECSubWrite(tid=71, reqid=osd.4.0:71, at_version=42'33, trim_to=0'0, roll_forward_to=37'31))", 2024-04-19T02:03:29.630 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.753652+0000", 2024-04-19T02:03:29.630 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 2.836445538, 2024-04-19T02:03:29.630 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012125554, 2024-04-19T02:03:29.630 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:29.630 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:29.630 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:29.630 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:29.630 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.630 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:29.630 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753652+0000", 2024-04-19T02:03:29.630 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.631 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.631 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.631 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:29.631 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753652+0000", 2024-04-19T02:03:29.631 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.631 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.631 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.631 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:29.631 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753654+0000", 2024-04-19T02:03:29.631 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.35e-06 2024-04-19T02:03:29.631 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.631 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.631 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:29.631 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753719+0000", 2024-04-19T02:03:29.631 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5501999999999997e-05 2024-04-19T02:03:29.632 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.632 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.632 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:29.632 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753747+0000", 2024-04-19T02:03:29.632 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7691e-05 2024-04-19T02:03:29.632 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.632 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.632 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:29.632 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753772+0000", 2024-04-19T02:03:29.632 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4825e-05 2024-04-19T02:03:29.632 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.632 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.632 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:29.632 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.754382+0000", 2024-04-19T02:03:29.632 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00061038900000000003 2024-04-19T02:03:29.633 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.633 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.633 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:29.633 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.756197+0000", 2024-04-19T02:03:29.633 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0018153589999999999 2024-04-19T02:03:29.633 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.633 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.633 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:29.633 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.765586+0000", 2024-04-19T02:03:29.633 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0093882529999999992 2024-04-19T02:03:29.633 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.633 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.633 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:29.633 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.765778+0000", 2024-04-19T02:03:29.633 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00019218499999999999 2024-04-19T02:03:29.633 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.634 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:29.634 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.634 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.634 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.634 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:377 3.cs0 3:3da93e5c:::smithi05930959-42:head [read 0~589824,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:29.634 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.759004+0000", 2024-04-19T02:03:29.634 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 2.8310938800000001, 2024-04-19T02:03:29.634 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.035358317, 2024-04-19T02:03:29.634 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:29.634 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:29.634 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:29.634 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:29.634 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:29.634 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:29.635 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 377 2024-04-19T02:03:29.635 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.635 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:29.635 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.635 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:29.635 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759004+0000", 2024-04-19T02:03:29.635 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.635 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.635 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.635 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:29.635 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759004+0000", 2024-04-19T02:03:29.635 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.635 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.635 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.635 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:29.636 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759008+0000", 2024-04-19T02:03:29.636 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9190000000000001e-06 2024-04-19T02:03:29.636 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.636 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.636 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:29.636 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759024+0000", 2024-04-19T02:03:29.636 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6195e-05 2024-04-19T02:03:29.636 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.636 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.636 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:29.636 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759058+0000", 2024-04-19T02:03:29.636 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4238e-05 2024-04-19T02:03:29.636 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.636 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.636 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:29.637 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759080+0000", 2024-04-19T02:03:29.637 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1237999999999999e-05 2024-04-19T02:03:29.637 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.637 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.637 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:29.637 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.771701+0000", 2024-04-19T02:03:29.637 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012621098000000001 2024-04-19T02:03:29.637 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.637 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.637 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:29.637 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.771949+0000", 2024-04-19T02:03:29.637 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00024814599999999998 2024-04-19T02:03:29.637 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.637 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.637 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:29.637 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.788299+0000", 2024-04-19T02:03:29.638 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016350138 2024-04-19T02:03:29.638 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.638 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.638 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:29.638 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.788497+0000", 2024-04-19T02:03:29.638 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00019762100000000001 2024-04-19T02:03:29.638 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.638 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.638 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:29.638 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.794362+0000", 2024-04-19T02:03:29.638 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0058657240000000001 2024-04-19T02:03:29.638 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.638 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:29.638 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.638 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.639 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.639 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:378 3.cs0 3:3da93e5c:::smithi05930959-42:head [read 0~589824,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:29.639 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.759134+0000", 2024-04-19T02:03:29.639 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 2.830964077, 2024-04-19T02:03:29.639 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.044044933000000001, 2024-04-19T02:03:29.639 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:29.639 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:29.639 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:29.639 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:29.639 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:29.639 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:29.639 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 378 2024-04-19T02:03:29.639 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.639 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:29.639 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.640 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:29.640 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759134+0000", 2024-04-19T02:03:29.640 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.640 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.640 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.640 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:29.640 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759134+0000", 2024-04-19T02:03:29.640 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.640 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.640 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.640 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:29.640 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759136+0000", 2024-04-19T02:03:29.640 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8029999999999999e-06 2024-04-19T02:03:29.640 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.640 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.641 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:29.641 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759145+0000", 2024-04-19T02:03:29.641 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5789999999999993e-06 2024-04-19T02:03:29.641 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.641 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.641 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:29.641 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759166+0000", 2024-04-19T02:03:29.641 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0815e-05 2024-04-19T02:03:29.641 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.641 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.641 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:29.641 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759187+0000", 2024-04-19T02:03:29.641 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1299000000000001e-05 2024-04-19T02:03:29.642 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.642 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.642 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:29.642 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.772621+0000", 2024-04-19T02:03:29.642 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013434032 2024-04-19T02:03:29.642 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.642 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.642 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:29.642 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.772918+0000", 2024-04-19T02:03:29.642 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00029657400000000002 2024-04-19T02:03:29.642 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.642 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.642 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:29.643 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.794802+0000", 2024-04-19T02:03:29.643 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021883785999999999 2024-04-19T02:03:29.643 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.643 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.643 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:29.643 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.795248+0000", 2024-04-19T02:03:29.643 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00044615099999999999 2024-04-19T02:03:29.643 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.643 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.643 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:29.643 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.803179+0000", 2024-04-19T02:03:29.643 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0079308940000000008 2024-04-19T02:03:29.643 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.643 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:29.644 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.644 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.644 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.644 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:379 3.cs0 3:3da93e5c:::smithi05930959-42:head [read 0~589824,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:29.644 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.759209+0000", 2024-04-19T02:03:29.644 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 2.83088834, 2024-04-19T02:03:29.644 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.045345979000000002, 2024-04-19T02:03:29.644 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:29.644 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:29.644 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:29.644 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:29.644 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:29.644 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:29.644 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 379 2024-04-19T02:03:29.644 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.645 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:29.645 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.645 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:29.645 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759209+0000", 2024-04-19T02:03:29.645 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.645 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.645 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.645 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:29.645 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759209+0000", 2024-04-19T02:03:29.645 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.645 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.645 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.645 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:29.645 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759210+0000", 2024-04-19T02:03:29.646 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9700000000000005e-07 2024-04-19T02:03:29.646 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.646 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.646 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:29.646 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759218+0000", 2024-04-19T02:03:29.646 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9589999999999995e-06 2024-04-19T02:03:29.646 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.646 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.646 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:29.646 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759236+0000", 2024-04-19T02:03:29.646 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7209000000000001e-05 2024-04-19T02:03:29.646 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.646 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.646 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:29.647 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759249+0000", 2024-04-19T02:03:29.647 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3956e-05 2024-04-19T02:03:29.647 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.647 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.647 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:29.647 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.773037+0000", 2024-04-19T02:03:29.647 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013787415000000001 2024-04-19T02:03:29.647 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.647 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.647 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:29.647 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.773251+0000", 2024-04-19T02:03:29.647 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021379099999999999 2024-04-19T02:03:29.647 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.647 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.647 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:29.648 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.797127+0000", 2024-04-19T02:03:29.648 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.023876128999999999 2024-04-19T02:03:29.648 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.648 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.648 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:29.648 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.797749+0000", 2024-04-19T02:03:29.648 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00062207600000000005 2024-04-19T02:03:29.648 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.648 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.648 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:29.648 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.804555+0000", 2024-04-19T02:03:29.648 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0068065469999999996 2024-04-19T02:03:29.648 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.648 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:29.648 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.649 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.649 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.649 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.ds6 42/26 ECSubWrite(tid=164, reqid=client.4509.0:363, at_version=42'15, trim_to=0'0, roll_forward_to=39'12))", 2024-04-19T02:03:29.649 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.760370+0000", 2024-04-19T02:03:29.649 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 2.82972767, 2024-04-19T02:03:29.649 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010134341999999999, 2024-04-19T02:03:29.649 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:29.649 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:29.649 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:29.649 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:29.649 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.649 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:29.649 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760370+0000", 2024-04-19T02:03:29.649 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.649 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.649 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.650 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:29.650 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760370+0000", 2024-04-19T02:03:29.650 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.650 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.650 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.650 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:29.650 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760372+0000", 2024-04-19T02:03:29.650 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0229999999999999e-06 2024-04-19T02:03:29.650 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.650 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.650 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:29.650 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760699+0000", 2024-04-19T02:03:29.650 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00032690099999999999 2024-04-19T02:03:29.651 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.651 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.651 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:29.651 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760733+0000", 2024-04-19T02:03:29.651 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3927000000000002e-05 2024-04-19T02:03:29.651 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.651 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.651 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:29.651 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760756+0000", 2024-04-19T02:03:29.651 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3292e-05 2024-04-19T02:03:29.651 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.651 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.651 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:29.651 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.763173+0000", 2024-04-19T02:03:29.651 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0024166389999999999 2024-04-19T02:03:29.652 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.652 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.652 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:29.652 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.763199+0000", 2024-04-19T02:03:29.652 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6010999999999999e-05 2024-04-19T02:03:29.652 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.652 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.652 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:29.652 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.770435+0000", 2024-04-19T02:03:29.652 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0072360549999999999 2024-04-19T02:03:29.652 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.652 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.652 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:29.652 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.770504+0000", 2024-04-19T02:03:29.652 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9493999999999996e-05 2024-04-19T02:03:29.653 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.653 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:29.653 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.653 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.653 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.653 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.5s3 42/26 ECSubWrite(tid=52, reqid=osd.10.0:52, at_version=42'29, trim_to=0'0, roll_forward_to=39'25))", 2024-04-19T02:03:29.653 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.760812+0000", 2024-04-19T02:03:29.653 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 2.8292854050000003, 2024-04-19T02:03:29.653 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010763234, 2024-04-19T02:03:29.653 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:29.653 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:29.653 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:29.653 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:29.653 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.653 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:29.653 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760812+0000", 2024-04-19T02:03:29.654 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.654 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.654 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.654 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:29.654 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760812+0000", 2024-04-19T02:03:29.654 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.654 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.654 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.654 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:29.654 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760814+0000", 2024-04-19T02:03:29.654 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7269999999999999e-06 2024-04-19T02:03:29.654 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.654 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.654 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:29.655 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760866+0000", 2024-04-19T02:03:29.655 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1858999999999999e-05 2024-04-19T02:03:29.655 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.655 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.655 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:29.655 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760897+0000", 2024-04-19T02:03:29.655 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1068000000000001e-05 2024-04-19T02:03:29.655 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.655 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.655 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:29.655 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760920+0000", 2024-04-19T02:03:29.655 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2588000000000001e-05 2024-04-19T02:03:29.655 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.655 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.655 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:29.655 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.765904+0000", 2024-04-19T02:03:29.655 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.004983889 2024-04-19T02:03:29.656 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.656 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.656 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:29.656 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.765931+0000", 2024-04-19T02:03:29.656 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7898e-05 2024-04-19T02:03:29.656 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.656 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.656 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:29.656 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.771493+0000", 2024-04-19T02:03:29.656 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0055615250000000003 2024-04-19T02:03:29.656 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.656 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.656 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:29.656 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.771576+0000", 2024-04-19T02:03:29.656 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2680000000000001e-05 2024-04-19T02:03:29.656 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.657 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:29.657 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.657 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.657 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.657 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.es7 42/26 ECSubWrite(tid=32, reqid=osd.7.0:32, at_version=42'25, trim_to=0'0, roll_forward_to=40'23))", 2024-04-19T02:03:29.657 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.761131+0000", 2024-04-19T02:03:29.657 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 2.8289669829999999, 2024-04-19T02:03:29.657 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010525209000000001, 2024-04-19T02:03:29.657 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:29.657 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:29.657 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:29.657 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:29.657 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.657 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:29.657 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761131+0000", 2024-04-19T02:03:29.657 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.658 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.658 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.658 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:29.658 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761131+0000", 2024-04-19T02:03:29.658 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.658 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.658 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.658 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:29.658 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761132+0000", 2024-04-19T02:03:29.658 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1699999999999997e-07 2024-04-19T02:03:29.658 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.658 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.658 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:29.658 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761187+0000", 2024-04-19T02:03:29.659 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5186000000000003e-05 2024-04-19T02:03:29.659 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.659 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.659 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:29.659 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761216+0000", 2024-04-19T02:03:29.659 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9213999999999998e-05 2024-04-19T02:03:29.659 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.659 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.659 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:29.659 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761238+0000", 2024-04-19T02:03:29.659 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2035000000000002e-05 2024-04-19T02:03:29.659 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.659 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.660 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:29.660 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761391+0000", 2024-04-19T02:03:29.660 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000152841 2024-04-19T02:03:29.660 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.660 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.660 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:29.660 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761417+0000", 2024-04-19T02:03:29.660 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6548000000000001e-05 2024-04-19T02:03:29.660 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.660 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.660 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:29.660 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.771598+0000", 2024-04-19T02:03:29.660 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010180829000000001 2024-04-19T02:03:29.660 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.661 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.661 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:29.661 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.771656+0000", 2024-04-19T02:03:29.661 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7839e-05 2024-04-19T02:03:29.661 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.661 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:29.661 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.661 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.661 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.661 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.ds6 42/26 ECSubWrite(tid=165, reqid=client.4509.0:364, at_version=42'16, trim_to=0'0, roll_forward_to=39'12))", 2024-04-19T02:03:29.661 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.774419+0000", 2024-04-19T02:03:29.661 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 2.8156785420000001, 2024-04-19T02:03:29.661 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0085658109999999996, 2024-04-19T02:03:29.661 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:29.661 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:29.661 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:29.662 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:29.662 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.662 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:29.662 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774419+0000", 2024-04-19T02:03:29.662 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.662 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.662 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.662 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:29.662 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774419+0000", 2024-04-19T02:03:29.662 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.662 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.662 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.662 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:29.662 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774421+0000", 2024-04-19T02:03:29.662 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3939999999999999e-06 2024-04-19T02:03:29.663 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.663 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.663 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:29.663 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774715+0000", 2024-04-19T02:03:29.663 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0002943 2024-04-19T02:03:29.663 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.663 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.663 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:29.663 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774744+0000", 2024-04-19T02:03:29.663 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9437999999999998e-05 2024-04-19T02:03:29.663 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.663 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.663 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:29.663 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774766+0000", 2024-04-19T02:03:29.663 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1404999999999999e-05 2024-04-19T02:03:29.663 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.664 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.664 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:29.664 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774907+0000", 2024-04-19T02:03:29.664 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000141655 2024-04-19T02:03:29.664 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.664 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.664 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:29.664 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774944+0000", 2024-04-19T02:03:29.664 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6216e-05 2024-04-19T02:03:29.664 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.664 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.664 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:29.664 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.782910+0000", 2024-04-19T02:03:29.664 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0079663189999999995 2024-04-19T02:03:29.664 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.664 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.664 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:29.665 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.782985+0000", 2024-04-19T02:03:29.665 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5084000000000001e-05 2024-04-19T02:03:29.665 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.665 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:29.665 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.665 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.665 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.665 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.es7 42/26 ECSubWrite(tid=33, reqid=osd.7.0:32, at_version=0'0, trim_to=0'0, roll_forward_to=42'25))", 2024-04-19T02:03:29.665 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.778130+0000", 2024-04-19T02:03:29.665 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 2.811967697, 2024-04-19T02:03:29.665 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0069547460000000004, 2024-04-19T02:03:29.665 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:29.665 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:29.665 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:29.665 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:29.665 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.665 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:29.666 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778130+0000", 2024-04-19T02:03:29.666 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.666 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.666 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.666 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:29.666 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778130+0000", 2024-04-19T02:03:29.666 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.666 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.666 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.666 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:29.666 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778132+0000", 2024-04-19T02:03:29.666 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.683e-06 2024-04-19T02:03:29.666 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.666 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.666 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:29.666 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778161+0000", 2024-04-19T02:03:29.667 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8911000000000002e-05 2024-04-19T02:03:29.667 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.667 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.667 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:29.667 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778184+0000", 2024-04-19T02:03:29.667 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3618e-05 2024-04-19T02:03:29.667 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.667 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.667 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:29.667 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778205+0000", 2024-04-19T02:03:29.667 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0477999999999999e-05 2024-04-19T02:03:29.667 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.667 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.667 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:29.667 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778366+0000", 2024-04-19T02:03:29.667 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000161082 2024-04-19T02:03:29.668 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.668 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.668 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:29.668 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778388+0000", 2024-04-19T02:03:29.668 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2515e-05 2024-04-19T02:03:29.668 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.668 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.668 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:29.668 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.785022+0000", 2024-04-19T02:03:29.668 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0066334640000000004 2024-04-19T02:03:29.668 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.668 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.668 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:29.668 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.785085+0000", 2024-04-19T02:03:29.668 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2995000000000002e-05 2024-04-19T02:03:29.668 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.668 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:29.669 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.669 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.669 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.669 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.8s5 42/26 ECSubWrite(tid=72, reqid=osd.4.0:71, at_version=0'0, trim_to=0'0, roll_forward_to=42'33))", 2024-04-19T02:03:29.669 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.778169+0000", 2024-04-19T02:03:29.669 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 2.8119290609999998, 2024-04-19T02:03:29.669 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0068147859999999998, 2024-04-19T02:03:29.669 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:29.669 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:29.669 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:29.669 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:29.669 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.669 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:29.669 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778169+0000", 2024-04-19T02:03:29.669 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.670 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.670 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.670 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:29.670 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778169+0000", 2024-04-19T02:03:29.670 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.670 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.670 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.670 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:29.670 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778171+0000", 2024-04-19T02:03:29.670 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8509999999999999e-06 2024-04-19T02:03:29.670 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.670 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.670 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:29.670 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778196+0000", 2024-04-19T02:03:29.670 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5894999999999999e-05 2024-04-19T02:03:29.671 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.671 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.671 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:29.671 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778226+0000", 2024-04-19T02:03:29.671 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9754999999999999e-05 2024-04-19T02:03:29.671 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.671 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.671 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:29.671 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778265+0000", 2024-04-19T02:03:29.671 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8699999999999999e-05 2024-04-19T02:03:29.671 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.671 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.671 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:29.671 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778889+0000", 2024-04-19T02:03:29.671 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00062377699999999995 2024-04-19T02:03:29.672 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.672 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.672 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:29.672 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778912+0000", 2024-04-19T02:03:29.672 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2952999999999999e-05 2024-04-19T02:03:29.672 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.672 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.672 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:29.672 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.784894+0000", 2024-04-19T02:03:29.672 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0059827650000000001 2024-04-19T02:03:29.672 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.672 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.672 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:29.672 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.784984+0000", 2024-04-19T02:03:29.672 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9090000000000003e-05 2024-04-19T02:03:29.672 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.673 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:29.673 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.673 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.673 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.673 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.as3 43/26 ECSubWrite(tid=89, reqid=client.4509.0:381, at_version=43'36, trim_to=0'0, roll_forward_to=42'34))", 2024-04-19T02:03:29.673 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:27.095497+0000", 2024-04-19T02:03:29.673 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 2.4946011530000001, 2024-04-19T02:03:29.673 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0084186580000000007, 2024-04-19T02:03:29.673 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:29.673 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:29.673 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:29.673 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:29.673 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.673 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:29.673 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.095497+0000", 2024-04-19T02:03:29.673 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.674 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.674 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.674 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:29.674 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.095497+0000", 2024-04-19T02:03:29.674 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:29.674 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.674 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.674 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:29.674 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.095497+0000", 2024-04-19T02:03:29.674 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3600000000000004e-07 2024-04-19T02:03:29.674 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.674 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.674 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:29.674 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.095643+0000", 2024-04-19T02:03:29.674 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014561899999999999 2024-04-19T02:03:29.675 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.675 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.675 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:29.675 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.095676+0000", 2024-04-19T02:03:29.675 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3531000000000001e-05 2024-04-19T02:03:29.675 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.675 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.675 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:29.675 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.095705+0000", 2024-04-19T02:03:29.675 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8691e-05 2024-04-19T02:03:29.675 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.675 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.675 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:29.675 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.095877+0000", 2024-04-19T02:03:29.675 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000171597 2024-04-19T02:03:29.675 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.675 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.676 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:29.676 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.095904+0000", 2024-04-19T02:03:29.676 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7487999999999998e-05 2024-04-19T02:03:29.676 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.676 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.676 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:29.676 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.103851+0000", 2024-04-19T02:03:29.676 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0079467579999999999 2024-04-19T02:03:29.676 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:29.676 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:29.676 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:29.676 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.103915+0000", 2024-04-19T02:03:29.676 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4437999999999995e-05 2024-04-19T02:03:29.676 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.676 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:29.676 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.676 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:29.677 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:29.677 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:29.678 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-04-19T02:03:29.679 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-19T02:03:29.680 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:29.678+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:29.680 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T02:03:29.778 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:29.778+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:29.847 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:29.847 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:03:29.848 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-04-19T02:03:29.848 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:29.861 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-19T02:03:29.905 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:29.906+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:29.980 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:29.978+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:30.025 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:30.025 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:03:30.025 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-04-19T02:03:30.025 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-04-19T02:03:30.025 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:30.039 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-19T02:03:30.049 INFO:teuthology.orchestra.run.smithi059.stderr:nodeep-scrub is unset 2024-04-19T02:03:30.050 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.3 to 1 (10000) 2024-04-19T02:03:30.062 INFO:tasks.rados.rados.0.smithi059.stdout:157: done (4 left) 2024-04-19T02:03:30.062 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 36 current snap is 9 2024-04-19T02:03:30.062 INFO:tasks.rados.rados.0.smithi059.stdout:158: seq_num 88 ranges {589824=720896,1310720=442368,1753088=327680} 2024-04-19T02:03:30.067 INFO:tasks.rados.rados.0.smithi059.stdout:158: writing smithi05930959-36 from 589824 to 1310720 tid 1 2024-04-19T02:03:30.071 INFO:tasks.rados.rados.0.smithi059.stdout:158: writing smithi05930959-36 from 1310720 to 1753088 tid 2 2024-04-19T02:03:30.074 INFO:tasks.rados.rados.0.smithi059.stdout:158: writing smithi05930959-36 from 1753088 to 2080768 tid 3 2024-04-19T02:03:30.081 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:30.082+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:30.082 INFO:tasks.rados.rados.0.smithi059.stdout:153: expect (ObjNum 78 snap 6 seq_num 78) 2024-04-19T02:03:30.092 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-19T02:03:30.152 INFO:tasks.rados.rados.0.smithi059.stdout:154: finishing write tid 1 to smithi05930959-41 2024-04-19T02:03:30.152 INFO:tasks.rados.rados.0.smithi059.stdout:154: finishing write tid 2 to smithi05930959-41 2024-04-19T02:03:30.152 INFO:tasks.rados.rados.0.smithi059.stdout:154: finishing write tid 3 to smithi05930959-41 2024-04-19T02:03:30.152 INFO:tasks.rados.rados.0.smithi059.stdout:154: finishing write tid 4 to smithi05930959-41 2024-04-19T02:03:30.152 INFO:tasks.rados.rados.0.smithi059.stdout:154: finishing write tid 5 to smithi05930959-41 2024-04-19T02:03:30.153 INFO:tasks.rados.rados.0.smithi059.stdout:154: oid 41 updating version 0 to 43 2024-04-19T02:03:30.153 INFO:tasks.rados.rados.0.smithi059.stdout:154: oid 41 version 43 is already newer than 42 2024-04-19T02:03:30.153 INFO:tasks.rados.rados.0.smithi059.stdout:154: oid 41 updating version 43 to 45 2024-04-19T02:03:30.153 INFO:tasks.rados.rados.0.smithi059.stdout:154: oid 41 version 45 is already newer than 44 2024-04-19T02:03:30.153 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 41 v 45 (ObjNum 86 snap 9 seq_num 86) dirty exists 2024-04-19T02:03:30.153 INFO:tasks.rados.rados.0.smithi059.stdout:154: left oid 41 (ObjNum 86 snap 9 seq_num 86) 2024-04-19T02:03:30.153 INFO:tasks.rados.rados.0.smithi059.stdout:155: expect (ObjNum 83 snap 8 seq_num 83) 2024-04-19T02:03:30.182 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:30.182+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:30.190 INFO:tasks.rados.rados.0.smithi059.stdout:156: finishing write tid 1 to smithi05930959-40 2024-04-19T02:03:30.190 INFO:tasks.rados.rados.0.smithi059.stdout:156: finishing write tid 2 to smithi05930959-40 2024-04-19T02:03:30.191 INFO:tasks.rados.rados.0.smithi059.stdout:156: finishing write tid 3 to smithi05930959-40 2024-04-19T02:03:30.191 INFO:tasks.rados.rados.0.smithi059.stdout:156: finishing write tid 4 to smithi05930959-40 2024-04-19T02:03:30.191 INFO:tasks.rados.rados.0.smithi059.stdout:156: finishing write tid 5 to smithi05930959-40 2024-04-19T02:03:30.191 INFO:tasks.rados.rados.0.smithi059.stdout:156: oid 40 updating version 0 to 39 2024-04-19T02:03:30.191 INFO:tasks.rados.rados.0.smithi059.stdout:156: oid 40 version 39 is already newer than 38 2024-04-19T02:03:30.191 INFO:tasks.rados.rados.0.smithi059.stdout:156: oid 40 updating version 39 to 40 2024-04-19T02:03:30.191 INFO:tasks.rados.rados.0.smithi059.stdout:156: oid 40 version 40 is already newer than 37 2024-04-19T02:03:30.191 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 40 v 40 (ObjNum 87 snap 9 seq_num 87) dirty exists 2024-04-19T02:03:30.191 INFO:tasks.rados.rados.0.smithi059.stdout:156: left oid 40 (ObjNum 87 snap 9 seq_num 87) 2024-04-19T02:03:30.191 INFO:tasks.rados.rados.0.smithi059.stdout:158: finishing write tid 1 to smithi05930959-36 2024-04-19T02:03:30.191 INFO:tasks.rados.rados.0.smithi059.stdout:158: finishing write tid 2 to smithi05930959-36 2024-04-19T02:03:30.191 INFO:tasks.rados.rados.0.smithi059.stdout:153: done (4 left) 2024-04-19T02:03:30.191 INFO:tasks.rados.rados.0.smithi059.stdout:154: done (3 left) 2024-04-19T02:03:30.191 INFO:tasks.rados.rados.0.smithi059.stdout:155: done (2 left) 2024-04-19T02:03:30.191 INFO:tasks.rados.rados.0.smithi059.stdout:156: done (1 left) 2024-04-19T02:03:30.192 INFO:tasks.rados.rados.0.smithi059.stdout:159: copy_from oid 30 from oid 32 current snap is 9 2024-04-19T02:03:30.192 INFO:tasks.rados.rados.0.smithi059.stdout:158: finishing write tid 3 to smithi05930959-36 2024-04-19T02:03:30.192 INFO:tasks.rados.rados.0.smithi059.stdout:158: finishing write tid 4 to smithi05930959-36 2024-04-19T02:03:30.192 INFO:tasks.rados.rados.0.smithi059.stdout:158: finishing write tid 5 to smithi05930959-36 2024-04-19T02:03:30.192 INFO:tasks.rados.rados.0.smithi059.stdout:158: oid 36 updating version 0 to 8 2024-04-19T02:03:30.192 INFO:tasks.rados.rados.0.smithi059.stdout:158: oid 36 updating version 8 to 10 2024-04-19T02:03:30.192 INFO:tasks.rados.rados.0.smithi059.stdout:158: oid 36 updating version 10 to 11 2024-04-19T02:03:30.192 INFO:tasks.rados.rados.0.smithi059.stdout:158: oid 36 version 11 is already newer than 9 2024-04-19T02:03:30.192 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 36 v 11 (ObjNum 88 snap 9 seq_num 88) dirty exists 2024-04-19T02:03:30.192 INFO:tasks.rados.rados.0.smithi059.stdout:158: left oid 36 (ObjNum 88 snap 9 seq_num 88) 2024-04-19T02:03:30.192 INFO:tasks.rados.rados.0.smithi059.stdout:158: done (1 left) 2024-04-19T02:03:30.192 INFO:tasks.rados.rados.0.smithi059.stdout:160: copy_from oid 10 from oid 36 current snap is 9 2024-04-19T02:03:30.192 INFO:tasks.rados.rados.0.smithi059.stdout:161: rollback oid 22 current snap is 9 2024-04-19T02:03:30.192 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 22 to 8 2024-04-19T02:03:30.192 INFO:tasks.rados.rados.0.smithi059.stdout:162: rollback oid 4 current snap is 9 2024-04-19T02:03:30.193 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 4 to 8 2024-04-19T02:03:30.193 INFO:tasks.rados.rados.0.smithi059.stdout:163: rollback oid 21 current snap is 9 2024-04-19T02:03:30.193 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 21 to 8 2024-04-19T02:03:30.193 INFO:tasks.rados.rados.0.smithi059.stdout:164: read oid 40 snap -1 2024-04-19T02:03:30.193 INFO:tasks.rados.rados.0.smithi059.stdout:164: expect (ObjNum 87 snap 9 seq_num 87) 2024-04-19T02:03:30.193 INFO:tasks.rados.rados.0.smithi059.stdout:165: read oid 41 snap -1 2024-04-19T02:03:30.193 INFO:tasks.rados.rados.0.smithi059.stdout:165: expect (ObjNum 86 snap 9 seq_num 86) 2024-04-19T02:03:30.193 INFO:tasks.rados.rados.0.smithi059.stdout:166: copy_from oid 43 from oid 17 current snap is 9 2024-04-19T02:03:30.193 INFO:tasks.rados.rados.0.smithi059.stdout:167: rollback oid 27 current snap is 9 2024-04-19T02:03:30.193 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 27 to 8 2024-04-19T02:03:30.193 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 5 current snap is 9 2024-04-19T02:03:30.193 INFO:tasks.rados.rados.0.smithi059.stdout:168: seq_num 89 ranges {81920=737280,819200=573440,1392640=196608} 2024-04-19T02:03:30.203 INFO:tasks.rados.rados.0.smithi059.stdout:168: writing smithi05930959-5 from 81920 to 819200 tid 1 2024-04-19T02:03:30.215 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:30.215 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-04-19T02:03:30.215 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-04-19T02:03:30.215 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-04-19T02:03:30.215 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.215 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s2 32/26 ECSubWrite(tid=76, reqid=client.4509.0:236, at_version=0'0, trim_to=0'0, roll_forward_to=32'29))", 2024-04-19T02:03:30.215 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:16.669478+0000", 2024-04-19T02:03:30.215 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 13.544052878, 2024-04-19T02:03:30.215 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0063499580000000002, 2024-04-19T02:03:30.216 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.216 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.216 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.216 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.216 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.216 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.216 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.669478+0000", 2024-04-19T02:03:30.216 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.216 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.216 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.216 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.216 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.669478+0000", 2024-04-19T02:03:30.216 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.216 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.216 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.217 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.217 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.669480+0000", 2024-04-19T02:03:30.217 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.59e-06 2024-04-19T02:03:30.217 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.217 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.217 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.217 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.669504+0000", 2024-04-19T02:03:30.217 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3770000000000001e-05 2024-04-19T02:03:30.217 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.217 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.217 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.217 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.669526+0000", 2024-04-19T02:03:30.218 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1872999999999999e-05 2024-04-19T02:03:30.218 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.218 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.218 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.218 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.669560+0000", 2024-04-19T02:03:30.218 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4813e-05 2024-04-19T02:03:30.218 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.218 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.218 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.218 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.669761+0000", 2024-04-19T02:03:30.218 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000200519 2024-04-19T02:03:30.218 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.218 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.218 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.218 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.669789+0000", 2024-04-19T02:03:30.219 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7776e-05 2024-04-19T02:03:30.219 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.219 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.219 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.219 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.675761+0000", 2024-04-19T02:03:30.219 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0059727909999999999 2024-04-19T02:03:30.219 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.219 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.219 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.219 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.675828+0000", 2024-04-19T02:03:30.219 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6826e-05 2024-04-19T02:03:30.219 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.219 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.219 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.219 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.219 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.220 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s2 32/26 ECSubWrite(tid=77, reqid=osd.3.0:77, at_version=32'31, trim_to=0'0, roll_forward_to=32'29))", 2024-04-19T02:03:30.220 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:16.672490+0000", 2024-04-19T02:03:30.220 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 13.541040775000001, 2024-04-19T02:03:30.220 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0051108880000000001, 2024-04-19T02:03:30.220 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.220 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.220 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.220 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.220 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.220 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.220 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.672490+0000", 2024-04-19T02:03:30.220 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.220 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.220 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.221 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.221 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.672490+0000", 2024-04-19T02:03:30.221 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.221 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.221 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.221 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.221 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.672492+0000", 2024-04-19T02:03:30.221 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5740000000000001e-06 2024-04-19T02:03:30.221 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.221 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.221 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.221 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.672538+0000", 2024-04-19T02:03:30.221 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6343e-05 2024-04-19T02:03:30.221 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.221 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.221 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.222 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.672580+0000", 2024-04-19T02:03:30.222 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1579999999999998e-05 2024-04-19T02:03:30.222 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.222 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.222 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.222 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.672604+0000", 2024-04-19T02:03:30.222 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4219e-05 2024-04-19T02:03:30.222 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.222 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.222 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.222 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.672766+0000", 2024-04-19T02:03:30.222 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016225500000000001 2024-04-19T02:03:30.222 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.223 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.223 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.223 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.672796+0000", 2024-04-19T02:03:30.223 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.991e-05 2024-04-19T02:03:30.223 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.223 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.223 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.223 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.677507+0000", 2024-04-19T02:03:30.223 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0047111410000000003 2024-04-19T02:03:30.223 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.223 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.223 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.223 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.677601+0000", 2024-04-19T02:03:30.224 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3865999999999999e-05 2024-04-19T02:03:30.224 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.224 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.224 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.224 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.224 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.224 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s2 32/26 ECSubWrite(tid=78, reqid=osd.3.0:78, at_version=32'33, trim_to=0'0, roll_forward_to=32'29))", 2024-04-19T02:03:30.224 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:16.674340+0000", 2024-04-19T02:03:30.224 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 13.539191312, 2024-04-19T02:03:30.224 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0058742509999999996, 2024-04-19T02:03:30.224 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.224 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.224 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.224 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.224 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.224 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.225 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.674340+0000", 2024-04-19T02:03:30.225 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.225 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.225 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.225 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.225 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.674340+0000", 2024-04-19T02:03:30.225 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.225 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.225 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.225 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.225 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.674342+0000", 2024-04-19T02:03:30.225 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.84e-06 2024-04-19T02:03:30.225 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.225 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.225 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.226 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.674395+0000", 2024-04-19T02:03:30.226 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3159999999999999e-05 2024-04-19T02:03:30.226 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.226 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.226 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.226 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.674441+0000", 2024-04-19T02:03:30.226 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5812000000000001e-05 2024-04-19T02:03:30.226 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.226 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.226 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.226 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.674466+0000", 2024-04-19T02:03:30.226 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5077000000000001e-05 2024-04-19T02:03:30.226 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.226 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.227 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.227 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.674631+0000", 2024-04-19T02:03:30.227 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016504900000000001 2024-04-19T02:03:30.227 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.227 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.227 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.227 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.674656+0000", 2024-04-19T02:03:30.227 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5069999999999999e-05 2024-04-19T02:03:30.227 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.227 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.227 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.227 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.680157+0000", 2024-04-19T02:03:30.227 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0055009580000000002 2024-04-19T02:03:30.227 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.227 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.228 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.228 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.680214+0000", 2024-04-19T02:03:30.228 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7284999999999998e-05 2024-04-19T02:03:30.228 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.228 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.228 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.228 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.228 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.228 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:293 3.5s0 3:a41c791e:::smithi05930959-15:head [delete] snapc 6=[6,5] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e37)", 2024-04-19T02:03:30.228 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:21.689881+0000", 2024-04-19T02:03:30.228 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 8.5236504699999998, 2024-04-19T02:03:30.228 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011022188, 2024-04-19T02:03:30.228 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.228 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.228 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:30.229 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:30.229 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:30.229 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:30.229 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 293 2024-04-19T02:03:30.229 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.229 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.229 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.229 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.229 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.689881+0000", 2024-04-19T02:03:30.229 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.229 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.229 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.229 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.229 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.689881+0000", 2024-04-19T02:03:30.230 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.230 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.230 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.230 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.230 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.689883+0000", 2024-04-19T02:03:30.230 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6170000000000001e-06 2024-04-19T02:03:30.230 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.230 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.230 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.230 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.689896+0000", 2024-04-19T02:03:30.230 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2826000000000001e-05 2024-04-19T02:03:30.230 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.230 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.230 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.231 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.689930+0000", 2024-04-19T02:03:30.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4289999999999999e-05 2024-04-19T02:03:30.231 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.231 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.231 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.689949+0000", 2024-04-19T02:03:30.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8921000000000001e-05 2024-04-19T02:03:30.231 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.231 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.231 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.690106+0000", 2024-04-19T02:03:30.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000156496 2024-04-19T02:03:30.231 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.231 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op must wait for map", 2024-04-19T02:03:30.231 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.690123+0000", 2024-04-19T02:03:30.232 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6949e-05 2024-04-19T02:03:30.232 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.232 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.232 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.232 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.693813+0000", 2024-04-19T02:03:30.232 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0036899419999999999 2024-04-19T02:03:30.232 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.232 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.232 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:30.232 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.693962+0000", 2024-04-19T02:03:30.232 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014913 2024-04-19T02:03:30.232 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.232 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.232 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.233 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.695407+0000", 2024-04-19T02:03:30.233 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001444823 2024-04-19T02:03:30.233 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.233 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.233 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.233 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.698101+0000", 2024-04-19T02:03:30.233 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002694288 2024-04-19T02:03:30.233 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.233 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.233 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:30.233 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.700773+0000", 2024-04-19T02:03:30.233 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0026718940000000002 2024-04-19T02:03:30.233 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.233 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.234 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.234 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.700903+0000", 2024-04-19T02:03:30.234 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013001199999999999 2024-04-19T02:03:30.234 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.234 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.234 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.234 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.234 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.234 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s2 40/26 ECSubWrite(tid=127, reqid=client.4509.0:349, at_version=40'61, trim_to=0'0, roll_forward_to=39'59))", 2024-04-19T02:03:30.234 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:24.769761+0000", 2024-04-19T02:03:30.234 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 5.4437705689999998, 2024-04-19T02:03:30.234 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0081110869999999995, 2024-04-19T02:03:30.234 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.234 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.235 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.235 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.235 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.235 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.235 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.769761+0000", 2024-04-19T02:03:30.235 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.235 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.235 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.235 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.235 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.769761+0000", 2024-04-19T02:03:30.235 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.235 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.235 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.235 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.235 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.769761+0000", 2024-04-19T02:03:30.236 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8999999999999996e-07 2024-04-19T02:03:30.236 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.236 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.236 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.771285+0000", 2024-04-19T02:03:30.236 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0015236189999999999 2024-04-19T02:03:30.236 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.236 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.236 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.771314+0000", 2024-04-19T02:03:30.236 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8991000000000001e-05 2024-04-19T02:03:30.236 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.236 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.236 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.771329+0000", 2024-04-19T02:03:30.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5628999999999999e-05 2024-04-19T02:03:30.237 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.237 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.237 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.237 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.771469+0000", 2024-04-19T02:03:30.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000139135 2024-04-19T02:03:30.237 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.237 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.237 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.237 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.771495+0000", 2024-04-19T02:03:30.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6775e-05 2024-04-19T02:03:30.237 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.237 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.237 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.237 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.777763+0000", 2024-04-19T02:03:30.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0062677399999999999 2024-04-19T02:03:30.238 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.238 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.238 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.238 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.777872+0000", 2024-04-19T02:03:30.238 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000108608 2024-04-19T02:03:30.238 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.238 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.238 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.238 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.238 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.238 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.4s2 42/26 ECSubWrite(tid=43, reqid=osd.1.0:43, at_version=42'30, trim_to=0'0, roll_forward_to=40'28))", 2024-04-19T02:03:30.238 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.738323+0000", 2024-04-19T02:03:30.238 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 3.4752086310000001, 2024-04-19T02:03:30.238 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0089071480000000001, 2024-04-19T02:03:30.238 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.239 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.239 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.239 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.239 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.239 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.738323+0000", 2024-04-19T02:03:30.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.239 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.239 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.239 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.738323+0000", 2024-04-19T02:03:30.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.239 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.239 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.240 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.738323+0000", 2024-04-19T02:03:30.240 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0699999999999996e-07 2024-04-19T02:03:30.240 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.240 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.240 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.240 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.738379+0000", 2024-04-19T02:03:30.240 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5263000000000003e-05 2024-04-19T02:03:30.240 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.240 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.240 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.240 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.738410+0000", 2024-04-19T02:03:30.240 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1744e-05 2024-04-19T02:03:30.240 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.240 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.240 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.241 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.738437+0000", 2024-04-19T02:03:30.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6378000000000001e-05 2024-04-19T02:03:30.241 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.241 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.241 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.241 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.738669+0000", 2024-04-19T02:03:30.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00023269799999999999 2024-04-19T02:03:30.241 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.241 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.241 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.241 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.738698+0000", 2024-04-19T02:03:30.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8260999999999999e-05 2024-04-19T02:03:30.241 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.241 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.241 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.242 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747031+0000", 2024-04-19T02:03:30.242 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0083331759999999994 2024-04-19T02:03:30.242 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.242 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.242 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.242 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747230+0000", 2024-04-19T02:03:30.242 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00019892099999999999 2024-04-19T02:03:30.242 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.242 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.242 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.242 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.242 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.242 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.4s2 42/26 ECSubWrite(tid=44, reqid=osd.1.0:44, at_version=42'32, trim_to=0'0, roll_forward_to=40'28))", 2024-04-19T02:03:30.242 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.739493+0000", 2024-04-19T02:03:30.243 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 3.4740379149999998, 2024-04-19T02:03:30.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015165089, 2024-04-19T02:03:30.243 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.243 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.243 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.243 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.243 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.243 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.243 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.739493+0000", 2024-04-19T02:03:30.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.243 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.243 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.243 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.243 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.739493+0000", 2024-04-19T02:03:30.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.243 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.244 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.244 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.739495+0000", 2024-04-19T02:03:30.244 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8729999999999999e-06 2024-04-19T02:03:30.244 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.244 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.244 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.739550+0000", 2024-04-19T02:03:30.244 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5080000000000001e-05 2024-04-19T02:03:30.244 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.244 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.244 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.739584+0000", 2024-04-19T02:03:30.244 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3396999999999999e-05 2024-04-19T02:03:30.244 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.245 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.245 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.739611+0000", 2024-04-19T02:03:30.245 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7733999999999999e-05 2024-04-19T02:03:30.245 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.245 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.245 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.245 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.740435+0000", 2024-04-19T02:03:30.245 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00082331600000000004 2024-04-19T02:03:30.245 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.245 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.245 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.245 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.740463+0000", 2024-04-19T02:03:30.245 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7991999999999999e-05 2024-04-19T02:03:30.245 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.245 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.245 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.246 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.754535+0000", 2024-04-19T02:03:30.246 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014072092 2024-04-19T02:03:30.246 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.246 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.246 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.246 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.754658+0000", 2024-04-19T02:03:30.246 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012360499999999999 2024-04-19T02:03:30.246 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.246 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.246 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.246 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.247 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s2 42/26 ECSubWrite(tid=132, reqid=osd.3.0:132, at_version=42'63, trim_to=0'0, roll_forward_to=40'61))", 2024-04-19T02:03:30.247 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.747482+0000", 2024-04-19T02:03:30.247 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 3.4660494760000002, 2024-04-19T02:03:30.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0077632200000000004, 2024-04-19T02:03:30.247 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.247 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.247 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.247 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747482+0000", 2024-04-19T02:03:30.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747482+0000", 2024-04-19T02:03:30.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747482+0000", 2024-04-19T02:03:30.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3300000000000002e-07 2024-04-19T02:03:30.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747538+0000", 2024-04-19T02:03:30.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6002999999999999e-05 2024-04-19T02:03:30.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747566+0000", 2024-04-19T02:03:30.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7668e-05 2024-04-19T02:03:30.249 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.249 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.249 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747592+0000", 2024-04-19T02:03:30.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5581000000000001e-05 2024-04-19T02:03:30.249 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.249 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.249 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747894+0000", 2024-04-19T02:03:30.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00030199300000000002 2024-04-19T02:03:30.250 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747954+0000", 2024-04-19T02:03:30.250 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0309000000000001e-05 2024-04-19T02:03:30.250 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755153+0000", 2024-04-19T02:03:30.250 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0071992369999999998 2024-04-19T02:03:30.250 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755245+0000", 2024-04-19T02:03:30.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1796e-05 2024-04-19T02:03:30.251 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.251 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.251 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.251 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.251 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.251 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.8s7 42/26 ECSubWrite(tid=71, reqid=osd.4.0:71, at_version=42'33, trim_to=0'0, roll_forward_to=37'31))", 2024-04-19T02:03:30.251 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.747600+0000", 2024-04-19T02:03:30.251 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 3.4659308119999999, 2024-04-19T02:03:30.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012073659, 2024-04-19T02:03:30.251 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.251 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.251 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.251 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.251 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.251 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.252 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747600+0000", 2024-04-19T02:03:30.252 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.252 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.252 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.252 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.252 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747600+0000", 2024-04-19T02:03:30.252 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.252 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.252 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.252 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.252 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747601+0000", 2024-04-19T02:03:30.252 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5400000000000001e-07 2024-04-19T02:03:30.252 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.252 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.252 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747647+0000", 2024-04-19T02:03:30.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6529000000000001e-05 2024-04-19T02:03:30.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747671+0000", 2024-04-19T02:03:30.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3425e-05 2024-04-19T02:03:30.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747702+0000", 2024-04-19T02:03:30.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1531e-05 2024-04-19T02:03:30.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.254 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749865+0000", 2024-04-19T02:03:30.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0021629150000000001 2024-04-19T02:03:30.254 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.254 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.254 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.254 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749891+0000", 2024-04-19T02:03:30.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5568e-05 2024-04-19T02:03:30.254 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.254 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.254 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.254 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759578+0000", 2024-04-19T02:03:30.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0096872139999999995 2024-04-19T02:03:30.254 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.254 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.254 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.255 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759674+0000", 2024-04-19T02:03:30.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5922999999999996e-05 2024-04-19T02:03:30.255 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.255 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.255 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.255 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.255 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.bs5 42/26 ECSubWrite(tid=133, reqid=osd.3.0:133, at_version=42'21, trim_to=0'0, roll_forward_to=39'19))", 2024-04-19T02:03:30.255 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.748033+0000", 2024-04-19T02:03:30.255 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 3.465498352, 2024-04-19T02:03:30.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012077704999999999, 2024-04-19T02:03:30.255 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.255 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.255 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.255 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.256 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748033+0000", 2024-04-19T02:03:30.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.256 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748033+0000", 2024-04-19T02:03:30.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.256 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748035+0000", 2024-04-19T02:03:30.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.75e-06 2024-04-19T02:03:30.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748087+0000", 2024-04-19T02:03:30.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2456000000000001e-05 2024-04-19T02:03:30.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748118+0000", 2024-04-19T02:03:30.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0728000000000001e-05 2024-04-19T02:03:30.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748141+0000", 2024-04-19T02:03:30.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2824e-05 2024-04-19T02:03:30.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748342+0000", 2024-04-19T02:03:30.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000201444 2024-04-19T02:03:30.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748366+0000", 2024-04-19T02:03:30.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4278999999999999e-05 2024-04-19T02:03:30.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.259 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.259 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760028+0000", 2024-04-19T02:03:30.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011662077 2024-04-19T02:03:30.259 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.259 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.259 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760111+0000", 2024-04-19T02:03:30.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2146999999999998e-05 2024-04-19T02:03:30.259 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.259 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.259 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.259 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.260 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.bs5 42/26 ECSubWrite(tid=135, reqid=osd.3.0:135, at_version=42'23, trim_to=0'0, roll_forward_to=39'19))", 2024-04-19T02:03:30.260 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.749533+0000", 2024-04-19T02:03:30.260 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 3.4639978020000002, 2024-04-19T02:03:30.260 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012068692000000001, 2024-04-19T02:03:30.260 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.260 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.260 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.260 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.260 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.260 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749533+0000", 2024-04-19T02:03:30.260 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.260 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.260 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.261 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749533+0000", 2024-04-19T02:03:30.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.261 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.261 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.261 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749535+0000", 2024-04-19T02:03:30.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7999999999999999e-06 2024-04-19T02:03:30.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.261 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.261 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.261 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749586+0000", 2024-04-19T02:03:30.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1202999999999997e-05 2024-04-19T02:03:30.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749618+0000", 2024-04-19T02:03:30.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1393999999999998e-05 2024-04-19T02:03:30.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749817+0000", 2024-04-19T02:03:30.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00019913799999999999 2024-04-19T02:03:30.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.757176+0000", 2024-04-19T02:03:30.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0073591949999999998 2024-04-19T02:03:30.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.757200+0000", 2024-04-19T02:03:30.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4388000000000001e-05 2024-04-19T02:03:30.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761527+0000", 2024-04-19T02:03:30.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0043261899999999997 2024-04-19T02:03:30.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761602+0000", 2024-04-19T02:03:30.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5383999999999995e-05 2024-04-19T02:03:30.264 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.264 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.264 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.264 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.264 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.264 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s2 42/26 ECSubWrite(tid=134, reqid=osd.3.0:134, at_version=42'65, trim_to=0'0, roll_forward_to=40'61))", 2024-04-19T02:03:30.264 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.751035+0000", 2024-04-19T02:03:30.264 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 3.4624959689999999, 2024-04-19T02:03:30.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0071011690000000001, 2024-04-19T02:03:30.264 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.264 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.264 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.264 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.264 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.265 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.265 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751035+0000", 2024-04-19T02:03:30.265 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.265 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.265 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.265 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.265 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751035+0000", 2024-04-19T02:03:30.265 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.265 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.265 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.265 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.265 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751044+0000", 2024-04-19T02:03:30.265 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5229999999999995e-06 2024-04-19T02:03:30.265 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.265 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.266 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.266 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751092+0000", 2024-04-19T02:03:30.266 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8217000000000003e-05 2024-04-19T02:03:30.266 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.266 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.266 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.266 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751127+0000", 2024-04-19T02:03:30.266 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5373000000000002e-05 2024-04-19T02:03:30.266 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.266 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.266 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.266 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751150+0000", 2024-04-19T02:03:30.266 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3011999999999999e-05 2024-04-19T02:03:30.266 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.266 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751335+0000", 2024-04-19T02:03:30.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00018431900000000001 2024-04-19T02:03:30.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751360+0000", 2024-04-19T02:03:30.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5751000000000001e-05 2024-04-19T02:03:30.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758036+0000", 2024-04-19T02:03:30.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0066759510000000003 2024-04-19T02:03:30.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.268 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.268 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758136+0000", 2024-04-19T02:03:30.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000100023 2024-04-19T02:03:30.268 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.268 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.268 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.268 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 42/26 ECSubWriteReply(tid=51, last_complete=42'27, committed=1, applied=1))", 2024-04-19T02:03:30.268 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.753536+0000", 2024-04-19T02:03:30.268 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 3.4599952190000001, 2024-04-19T02:03:30.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0092102, 2024-04-19T02:03:30.268 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.268 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.268 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.269 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.269 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.269 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753536+0000", 2024-04-19T02:03:30.269 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.269 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.269 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753536+0000", 2024-04-19T02:03:30.269 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.269 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.269 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753538+0000", 2024-04-19T02:03:30.269 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.863e-06 2024-04-19T02:03:30.270 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.270 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.270 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.270 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753550+0000", 2024-04-19T02:03:30.270 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2102e-05 2024-04-19T02:03:30.270 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.270 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.270 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.270 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753580+0000", 2024-04-19T02:03:30.270 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0343999999999999e-05 2024-04-19T02:03:30.270 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.270 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.270 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.270 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753601+0000", 2024-04-19T02:03:30.270 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0499000000000001e-05 2024-04-19T02:03:30.271 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.271 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.271 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.271 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.762682+0000", 2024-04-19T02:03:30.271 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0090811969999999992 2024-04-19T02:03:30.271 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.271 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.271 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.271 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.762746+0000", 2024-04-19T02:03:30.271 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4195000000000005e-05 2024-04-19T02:03:30.271 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.271 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.271 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.271 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.272 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.272 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 42/26 ECSubWriteReply(tid=51, last_complete=42'27, committed=1, applied=1))", 2024-04-19T02:03:30.272 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.753740+0000", 2024-04-19T02:03:30.272 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 3.4597916519999998, 2024-04-19T02:03:30.272 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0091385830000000005, 2024-04-19T02:03:30.272 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.272 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.272 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.272 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.272 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.272 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.272 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753740+0000", 2024-04-19T02:03:30.272 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.272 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.272 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.273 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.273 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753740+0000", 2024-04-19T02:03:30.273 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.273 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.273 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.273 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.273 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753741+0000", 2024-04-19T02:03:30.273 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3030000000000001e-06 2024-04-19T02:03:30.273 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.273 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.273 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.273 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753754+0000", 2024-04-19T02:03:30.273 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3079e-05 2024-04-19T02:03:30.273 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.273 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.274 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.274 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753777+0000", 2024-04-19T02:03:30.274 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2841999999999999e-05 2024-04-19T02:03:30.274 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.274 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.274 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.274 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753796+0000", 2024-04-19T02:03:30.274 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9445000000000001e-05 2024-04-19T02:03:30.274 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.274 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.274 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.274 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.762824+0000", 2024-04-19T02:03:30.274 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0090278279999999999 2024-04-19T02:03:30.274 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.275 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.762878+0000", 2024-04-19T02:03:30.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4085999999999997e-05 2024-04-19T02:03:30.275 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.275 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.275 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.275 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 42/26 ECSubWriteReply(tid=51, last_complete=42'27, committed=1, applied=1))", 2024-04-19T02:03:30.275 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.754901+0000", 2024-04-19T02:03:30.275 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 3.458629889, 2024-04-19T02:03:30.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0080924169999999993, 2024-04-19T02:03:30.275 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.275 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.276 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.276 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.276 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.276 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.276 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.754901+0000", 2024-04-19T02:03:30.276 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.276 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.276 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.276 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.276 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.754901+0000", 2024-04-19T02:03:30.276 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.276 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.276 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.276 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.276 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.754910+0000", 2024-04-19T02:03:30.277 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4929999999999998e-06 2024-04-19T02:03:30.277 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.277 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.277 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.277 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.754921+0000", 2024-04-19T02:03:30.277 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1409e-05 2024-04-19T02:03:30.277 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.277 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.277 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.277 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.754949+0000", 2024-04-19T02:03:30.277 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7758000000000001e-05 2024-04-19T02:03:30.277 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.277 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.277 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.277 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.754967+0000", 2024-04-19T02:03:30.278 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8345000000000001e-05 2024-04-19T02:03:30.278 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.278 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.278 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.278 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.762949+0000", 2024-04-19T02:03:30.278 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0079821560000000007 2024-04-19T02:03:30.278 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.278 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.278 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.278 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.762994+0000", 2024-04-19T02:03:30.278 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4255999999999998e-05 2024-04-19T02:03:30.278 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.278 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.278 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.278 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.279 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.279 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.4s2 42/26 ECSubWrite(tid=46, reqid=client.4509.0:374, at_version=42'34, trim_to=0'0, roll_forward_to=42'32))", 2024-04-19T02:03:30.279 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.763360+0000", 2024-04-19T02:03:30.279 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 3.4501712169999998, 2024-04-19T02:03:30.279 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0082147320000000006, 2024-04-19T02:03:30.279 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.279 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.279 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.279 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.279 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.279 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.279 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.763360+0000", 2024-04-19T02:03:30.279 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.279 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.279 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.279 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.280 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.763360+0000", 2024-04-19T02:03:30.280 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.280 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.280 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.280 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.280 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.763361+0000", 2024-04-19T02:03:30.280 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.9999999999999995e-07 2024-04-19T02:03:30.280 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.280 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.280 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.280 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.763410+0000", 2024-04-19T02:03:30.280 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9440000000000001e-05 2024-04-19T02:03:30.280 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.280 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.280 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.763434+0000", 2024-04-19T02:03:30.281 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3754e-05 2024-04-19T02:03:30.281 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.281 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.281 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.763452+0000", 2024-04-19T02:03:30.281 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8332999999999999e-05 2024-04-19T02:03:30.281 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.281 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.281 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.763770+0000", 2024-04-19T02:03:30.281 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00031714500000000001 2024-04-19T02:03:30.281 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.281 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.281 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.282 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.763796+0000", 2024-04-19T02:03:30.282 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6451000000000001e-05 2024-04-19T02:03:30.282 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.282 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.282 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.282 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.771475+0000", 2024-04-19T02:03:30.282 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0076787069999999999 2024-04-19T02:03:30.282 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.282 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.282 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.282 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.771575+0000", 2024-04-19T02:03:30.282 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.9902000000000007e-05 2024-04-19T02:03:30.282 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.282 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.282 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.283 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.283 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.283 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.4s2 42/26 ECSubWrite(tid=47, reqid=client.4509.0:375, at_version=42'35, trim_to=0'0, roll_forward_to=42'32))", 2024-04-19T02:03:30.283 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.763496+0000", 2024-04-19T02:03:30.283 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 3.4500350270000002, 2024-04-19T02:03:30.283 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0081589360000000003, 2024-04-19T02:03:30.283 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.283 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.283 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.283 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.283 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.283 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.283 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.763496+0000", 2024-04-19T02:03:30.283 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.284 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.284 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.284 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.284 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.763496+0000", 2024-04-19T02:03:30.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.284 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.284 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.284 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.284 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.763497+0000", 2024-04-19T02:03:30.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7000000000000004e-07 2024-04-19T02:03:30.284 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.284 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.284 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.284 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.763542+0000", 2024-04-19T02:03:30.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5522999999999998e-05 2024-04-19T02:03:30.284 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.285 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.285 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.285 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.763565+0000", 2024-04-19T02:03:30.285 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2455000000000001e-05 2024-04-19T02:03:30.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.285 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.285 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.285 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.763582+0000", 2024-04-19T02:03:30.285 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7549000000000001e-05 2024-04-19T02:03:30.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.285 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.285 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.285 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.765476+0000", 2024-04-19T02:03:30.285 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0018940020000000001 2024-04-19T02:03:30.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.286 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.286 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.286 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.765502+0000", 2024-04-19T02:03:30.286 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5525e-05 2024-04-19T02:03:30.286 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.286 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.286 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.286 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.771591+0000", 2024-04-19T02:03:30.286 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0060889639999999997 2024-04-19T02:03:30.286 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.286 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.286 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.286 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.771655+0000", 2024-04-19T02:03:30.286 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4247999999999999e-05 2024-04-19T02:03:30.287 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.287 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.287 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.287 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.287 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.287 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.4s2 42/26 ECSubWrite(tid=48, reqid=client.4509.0:376, at_version=42'36, trim_to=0'0, roll_forward_to=42'32))", 2024-04-19T02:03:30.287 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.765260+0000", 2024-04-19T02:03:30.287 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 3.448270854, 2024-04-19T02:03:30.287 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0064638409999999997, 2024-04-19T02:03:30.287 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.287 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.287 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.287 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.287 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.287 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.288 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.765260+0000", 2024-04-19T02:03:30.288 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.288 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.288 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.288 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.288 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.765260+0000", 2024-04-19T02:03:30.288 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.288 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.288 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.288 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.288 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.765262+0000", 2024-04-19T02:03:30.288 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.736e-06 2024-04-19T02:03:30.288 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.289 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.289 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.289 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.765307+0000", 2024-04-19T02:03:30.289 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4634e-05 2024-04-19T02:03:30.289 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.289 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.289 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.289 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.765335+0000", 2024-04-19T02:03:30.289 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8347000000000002e-05 2024-04-19T02:03:30.289 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.289 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.289 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.289 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.765357+0000", 2024-04-19T02:03:30.289 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1939000000000001e-05 2024-04-19T02:03:30.289 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.290 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.290 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.290 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.766493+0000", 2024-04-19T02:03:30.290 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001136361 2024-04-19T02:03:30.290 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.290 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.290 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.290 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.766520+0000", 2024-04-19T02:03:30.290 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6182000000000001e-05 2024-04-19T02:03:30.290 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.290 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.290 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.290 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.771660+0000", 2024-04-19T02:03:30.290 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0051409669999999998 2024-04-19T02:03:30.291 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.291 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.291 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.291 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.771724+0000", 2024-04-19T02:03:30.291 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3675000000000003e-05 2024-04-19T02:03:30.291 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.291 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.291 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.291 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.291 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.291 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.bs5 42/26 ECSubWrite(tid=139, reqid=osd.3.0:135, at_version=0'0, trim_to=0'0, roll_forward_to=42'23))", 2024-04-19T02:03:30.291 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.774488+0000", 2024-04-19T02:03:30.291 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 3.4390430279999999, 2024-04-19T02:03:30.291 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0065933809999999997, 2024-04-19T02:03:30.292 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.292 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.292 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.292 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.292 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.292 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.292 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774488+0000", 2024-04-19T02:03:30.292 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.292 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.292 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.292 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.292 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774488+0000", 2024-04-19T02:03:30.292 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.292 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.292 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.293 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.293 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774490+0000", 2024-04-19T02:03:30.293 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1129999999999999e-06 2024-04-19T02:03:30.293 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.293 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.293 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.293 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774521+0000", 2024-04-19T02:03:30.293 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1015e-05 2024-04-19T02:03:30.293 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.293 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.293 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.293 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774565+0000", 2024-04-19T02:03:30.293 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3673e-05 2024-04-19T02:03:30.293 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.294 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.294 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.294 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774590+0000", 2024-04-19T02:03:30.294 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4691000000000001e-05 2024-04-19T02:03:30.294 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.294 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.294 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.294 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774746+0000", 2024-04-19T02:03:30.294 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015644000000000001 2024-04-19T02:03:30.294 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.294 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.294 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.294 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774776+0000", 2024-04-19T02:03:30.294 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0100999999999999e-05 2024-04-19T02:03:30.294 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.295 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.295 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.295 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.780988+0000", 2024-04-19T02:03:30.295 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0062118820000000002 2024-04-19T02:03:30.295 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.295 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.295 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.295 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.781082+0000", 2024-04-19T02:03:30.295 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3466000000000003e-05 2024-04-19T02:03:30.295 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.295 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.295 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.295 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.295 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.296 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.8s7 42/26 ECSubWrite(tid=72, reqid=osd.4.0:71, at_version=0'0, trim_to=0'0, roll_forward_to=42'33))", 2024-04-19T02:03:30.296 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.778603+0000", 2024-04-19T02:03:30.296 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 3.4349284419999999, 2024-04-19T02:03:30.296 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0051012840000000002, 2024-04-19T02:03:30.296 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.296 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.296 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.296 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.296 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.296 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.296 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778603+0000", 2024-04-19T02:03:30.296 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.296 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.297 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.297 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.297 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778603+0000", 2024-04-19T02:03:30.297 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.297 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.297 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.297 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.297 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778605+0000", 2024-04-19T02:03:30.297 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.897e-06 2024-04-19T02:03:30.297 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.297 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.297 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.297 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778634+0000", 2024-04-19T02:03:30.297 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9054999999999999e-05 2024-04-19T02:03:30.297 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.298 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.298 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.298 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778678+0000", 2024-04-19T02:03:30.298 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4353e-05 2024-04-19T02:03:30.298 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.298 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.298 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.298 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778702+0000", 2024-04-19T02:03:30.298 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4218000000000001e-05 2024-04-19T02:03:30.298 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.298 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.298 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.298 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778869+0000", 2024-04-19T02:03:30.298 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000167102 2024-04-19T02:03:30.298 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.299 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778893+0000", 2024-04-19T02:03:30.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3672999999999998e-05 2024-04-19T02:03:30.299 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.299 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.783583+0000", 2024-04-19T02:03:30.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0046902460000000003 2024-04-19T02:03:30.299 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.299 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.783704+0000", 2024-04-19T02:03:30.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012074 2024-04-19T02:03:30.299 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.299 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.300 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.300 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.300 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.300 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:30.302 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-19T02:03:30.303 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:30.302+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:30.303 INFO:tasks.rados.rados.0.smithi059.stdout:168: writing smithi05930959-5 from 819200 to 1392640 tid 2 2024-04-19T02:03:30.303 INFO:tasks.rados.rados.0.smithi059.stdout:168: writing smithi05930959-5 from 1392640 to 1589248 tid 3 2024-04-19T02:03:30.303 INFO:tasks.rados.rados.0.smithi059.stdout:163: finishing rollback tid 1 to smithi05930959-21 2024-04-19T02:03:30.303 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 21 v 0 (ObjNum 465 snap 0 seq_num 1243749840) dirty dne 2024-04-19T02:03:30.303 INFO:tasks.rados.rados.0.smithi059.stdout:162: finishing rollback tid 0 to smithi05930959-4 2024-04-19T02:03:30.303 INFO:tasks.rados.rados.0.smithi059.stdout:162: finishing rollback tid 1 to smithi05930959-4 2024-04-19T02:03:30.303 INFO:tasks.rados.rados.0.smithi059.stdout:162: finishing rollback tid 2 to smithi05930959-4 2024-04-19T02:03:30.303 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 4 v 38 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-19T02:03:30.303 INFO:tasks.rados.rados.0.smithi059.stdout:162: done (9 left) 2024-04-19T02:03:30.303 INFO:tasks.rados.rados.0.smithi059.stdout:163: done (8 left) 2024-04-19T02:03:30.303 INFO:tasks.rados.rados.0.smithi059.stdout:169: delete oid 33 current snap is 9 2024-04-19T02:03:30.304 INFO:tasks.rados.rados.0.smithi059.stdout:168: finishing write tid 1 to smithi05930959-5 2024-04-19T02:03:30.304 INFO:tasks.rados.rados.0.smithi059.stdout:164: expect (ObjNum 87 snap 9 seq_num 87) 2024-04-19T02:03:30.353 INFO:tasks.rados.rados.0.smithi059.stdout:168: finishing write tid 2 to smithi05930959-5 2024-04-19T02:03:30.353 INFO:tasks.rados.rados.0.smithi059.stdout:165: expect (ObjNum 86 snap 9 seq_num 86) 2024-04-19T02:03:30.402 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:30.402+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:30.471 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:30.471 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:03:30.471 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-04-19T02:03:30.471 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:30.482 INFO:tasks.rados.rados.0.smithi059.stdout:168: finishing write tid 3 to smithi05930959-5 2024-04-19T02:03:30.482 INFO:tasks.rados.rados.0.smithi059.stdout:168: finishing write tid 4 to smithi05930959-5 2024-04-19T02:03:30.482 INFO:tasks.rados.rados.0.smithi059.stdout:160: finishing copy_from racing read to smithi05930959-10 2024-04-19T02:03:30.482 INFO:tasks.rados.rados.0.smithi059.stdout:161: finishing rollback tid 0 to smithi05930959-22 2024-04-19T02:03:30.482 INFO:tasks.rados.rados.0.smithi059.stdout:168: finishing write tid 5 to smithi05930959-5 2024-04-19T02:03:30.483 INFO:tasks.rados.rados.0.smithi059.stdout:168: oid 5 updating version 0 to 32 2024-04-19T02:03:30.483 INFO:tasks.rados.rados.0.smithi059.stdout:168: oid 5 version 32 is already newer than 29 2024-04-19T02:03:30.483 INFO:tasks.rados.rados.0.smithi059.stdout:168: oid 5 version 32 is already newer than 30 2024-04-19T02:03:30.483 INFO:tasks.rados.rados.0.smithi059.stdout:168: oid 5 version 32 is already newer than 31 2024-04-19T02:03:30.483 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 5 v 32 (ObjNum 89 snap 9 seq_num 89) dirty exists 2024-04-19T02:03:30.483 INFO:tasks.rados.rados.0.smithi059.stdout:168: left oid 5 (ObjNum 89 snap 9 seq_num 89) 2024-04-19T02:03:30.483 INFO:tasks.rados.rados.0.smithi059.stdout:161: finishing rollback tid 1 to smithi05930959-22 2024-04-19T02:03:30.483 INFO:tasks.rados.rados.0.smithi059.stdout:161: finishing rollback tid 2 to smithi05930959-22 2024-04-19T02:03:30.483 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 22 v 79 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-19T02:03:30.483 INFO:tasks.rados.rados.0.smithi059.stdout:167: finishing rollback tid 0 to smithi05930959-27 2024-04-19T02:03:30.483 INFO:tasks.rados.rados.0.smithi059.stdout:167: finishing rollback tid 1 to smithi05930959-27 2024-04-19T02:03:30.483 INFO:tasks.rados.rados.0.smithi059.stdout:167: finishing rollback tid 2 to smithi05930959-27 2024-04-19T02:03:30.483 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 27 v 83 (ObjNum 79 snap 6 seq_num 79) dirty exists 2024-04-19T02:03:30.484 INFO:tasks.rados.rados.0.smithi059.stdout:160: finishing copy_from to smithi05930959-10 2024-04-19T02:03:30.484 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 10 v 11 (ObjNum 88 snap 9 seq_num 88) dirty exists 2024-04-19T02:03:30.484 INFO:tasks.rados.rados.0.smithi059.stdout:166: finishing copy_from racing read to smithi05930959-43 2024-04-19T02:03:30.484 INFO:tasks.rados.rados.0.smithi059.stdout:159: finishing copy_from racing read to smithi05930959-30 2024-04-19T02:03:30.484 INFO:tasks.rados.rados.0.smithi059.stdout:159: finishing copy_from to smithi05930959-30 2024-04-19T02:03:30.484 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 30 v 14 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-19T02:03:30.484 INFO:tasks.rados.rados.0.smithi059.stdout:166: finishing copy_from to smithi05930959-43 2024-04-19T02:03:30.484 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 43 v 51 (ObjNum 74 snap 6 seq_num 74) dirty exists 2024-04-19T02:03:30.484 INFO:tasks.rados.rados.0.smithi059.stdout:159: done (8 left) 2024-04-19T02:03:30.484 INFO:tasks.rados.rados.0.smithi059.stdout:160: done (7 left) 2024-04-19T02:03:30.484 INFO:tasks.rados.rados.0.smithi059.stdout:161: done (6 left) 2024-04-19T02:03:30.484 INFO:tasks.rados.rados.0.smithi059.stdout:164: done (5 left) 2024-04-19T02:03:30.484 INFO:tasks.rados.rados.0.smithi059.stdout:165: done (4 left) 2024-04-19T02:03:30.484 INFO:tasks.rados.rados.0.smithi059.stdout:166: done (3 left) 2024-04-19T02:03:30.484 INFO:tasks.rados.rados.0.smithi059.stdout:167: done (2 left) 2024-04-19T02:03:30.485 INFO:tasks.rados.rados.0.smithi059.stdout:168: done (1 left) 2024-04-19T02:03:30.485 INFO:tasks.rados.rados.0.smithi059.stdout:169: done (0 left) 2024-04-19T02:03:30.485 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 7 current snap is 9 2024-04-19T02:03:30.485 INFO:tasks.rados.rados.0.smithi059.stdout:170: seq_num 91 ranges {1343488=770048,2113536=475136,2588672=294912} 2024-04-19T02:03:30.486 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-19T02:03:30.515 INFO:tasks.rados.rados.0.smithi059.stdout:170: writing smithi05930959-7 from 1343488 to 2113536 tid 1 2024-04-19T02:03:30.521 INFO:tasks.rados.rados.0.smithi059.stdout:170: writing smithi05930959-7 from 2113536 to 2588672 tid 2 2024-04-19T02:03:30.526 INFO:tasks.rados.rados.0.smithi059.stdout:170: writing smithi05930959-7 from 2588672 to 2883584 tid 3 2024-04-19T02:03:30.526 INFO:tasks.rados.rados.0.smithi059.stdout:170: finishing write tid 1 to smithi05930959-7 2024-04-19T02:03:30.526 INFO:tasks.rados.rados.0.smithi059.stdout:171: read oid 25 snap -1 2024-04-19T02:03:30.526 INFO:tasks.rados.rados.0.smithi059.stdout:171: expect (ObjNum 52 snap 0 seq_num 52) 2024-04-19T02:03:30.526 INFO:tasks.rados.rados.0.smithi059.stdout:172: snap_remove snap 8 2024-04-19T02:03:30.533 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:30.534+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:30.604 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:30.602+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:30.609 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:30.609 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:03:30.610 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-04-19T02:03:30.610 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-04-19T02:03:30.610 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:30.625 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-19T02:03:30.705 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:30.706+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:30.789 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:30.789 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-04-19T02:03:30.789 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-04-19T02:03:30.789 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-04-19T02:03:30.790 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.790 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/26 ECSubWrite(tid=132, reqid=osd.3.0:132, at_version=42'63, trim_to=0'0, roll_forward_to=40'61))", 2024-04-19T02:03:30.790 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.748602+0000", 2024-04-19T02:03:30.790 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4.04321631, 2024-04-19T02:03:30.790 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.024032777000000002, 2024-04-19T02:03:30.790 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.790 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.790 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.790 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.790 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.790 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.790 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748602+0000", 2024-04-19T02:03:30.790 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.790 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.791 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.791 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.791 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748602+0000", 2024-04-19T02:03:30.791 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.791 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.791 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.791 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.791 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748603+0000", 2024-04-19T02:03:30.791 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4600000000000003e-07 2024-04-19T02:03:30.793 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.793 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.793 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.794 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748657+0000", 2024-04-19T02:03:30.794 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3803e-05 2024-04-19T02:03:30.794 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.794 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.794 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.794 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748688+0000", 2024-04-19T02:03:30.794 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0871000000000003e-05 2024-04-19T02:03:30.794 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.794 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.794 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.794 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748710+0000", 2024-04-19T02:03:30.794 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2634999999999999e-05 2024-04-19T02:03:30.794 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.795 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.795 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.795 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748980+0000", 2024-04-19T02:03:30.795 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00026984900000000001 2024-04-19T02:03:30.795 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.795 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.795 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.795 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749007+0000", 2024-04-19T02:03:30.795 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6225e-05 2024-04-19T02:03:30.795 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.795 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.795 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.795 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.772533+0000", 2024-04-19T02:03:30.795 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.023526782 2024-04-19T02:03:30.795 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.796 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.796 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.796 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.772635+0000", 2024-04-19T02:03:30.796 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000101766 2024-04-19T02:03:30.796 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.796 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.796 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.796 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.796 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.796 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.8s3 42/26 ECSubWrite(tid=71, reqid=osd.4.0:71, at_version=42'33, trim_to=0'0, roll_forward_to=37'31))", 2024-04-19T02:03:30.796 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.751483+0000", 2024-04-19T02:03:30.796 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4.0403358579999997, 2024-04-19T02:03:30.796 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.024697106999999999, 2024-04-19T02:03:30.796 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.797 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.797 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.797 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.797 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.797 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.797 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751483+0000", 2024-04-19T02:03:30.797 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.797 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.797 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.797 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.797 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751483+0000", 2024-04-19T02:03:30.797 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.797 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.797 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.797 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.798 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751483+0000", 2024-04-19T02:03:30.798 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9999999999999997e-07 2024-04-19T02:03:30.798 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.798 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.798 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.798 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751545+0000", 2024-04-19T02:03:30.798 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1958999999999994e-05 2024-04-19T02:03:30.798 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.798 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.798 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.798 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755035+0000", 2024-04-19T02:03:30.798 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0034897819999999999 2024-04-19T02:03:30.798 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.798 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.799 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.799 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755064+0000", 2024-04-19T02:03:30.799 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9227999999999999e-05 2024-04-19T02:03:30.799 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.799 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.799 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.799 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755228+0000", 2024-04-19T02:03:30.799 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016367700000000001 2024-04-19T02:03:30.799 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.799 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.799 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.799 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755256+0000", 2024-04-19T02:03:30.799 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7900999999999999e-05 2024-04-19T02:03:30.799 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.799 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.799 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.800 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.776106+0000", 2024-04-19T02:03:30.800 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020850560000000001 2024-04-19T02:03:30.800 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.800 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.800 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.800 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.776180+0000", 2024-04-19T02:03:30.800 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3399999999999995e-05 2024-04-19T02:03:30.800 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.800 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.800 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.800 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.800 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.800 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/26 ECSubWrite(tid=134, reqid=osd.3.0:134, at_version=42'65, trim_to=0'0, roll_forward_to=40'61))", 2024-04-19T02:03:30.800 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.751576+0000", 2024-04-19T02:03:30.800 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4.0402421510000002, 2024-04-19T02:03:30.800 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.027269287, 2024-04-19T02:03:30.801 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.801 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.801 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.801 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.801 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.801 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.801 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751576+0000", 2024-04-19T02:03:30.801 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.801 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.801 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.801 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.801 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751576+0000", 2024-04-19T02:03:30.801 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.801 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.801 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.801 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.801 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751578+0000", 2024-04-19T02:03:30.802 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9659999999999999e-06 2024-04-19T02:03:30.802 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.802 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.802 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.802 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751651+0000", 2024-04-19T02:03:30.802 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2781000000000007e-05 2024-04-19T02:03:30.802 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.802 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.802 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.802 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755528+0000", 2024-04-19T02:03:30.802 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0038770359999999999 2024-04-19T02:03:30.802 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.802 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.803 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.803 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755558+0000", 2024-04-19T02:03:30.803 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9414e-05 2024-04-19T02:03:30.803 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.803 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.803 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.803 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755853+0000", 2024-04-19T02:03:30.803 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00029508699999999998 2024-04-19T02:03:30.803 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.803 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.803 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.803 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755881+0000", 2024-04-19T02:03:30.803 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7965000000000001e-05 2024-04-19T02:03:30.803 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.803 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.804 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.804 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778764+0000", 2024-04-19T02:03:30.804 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022883244000000001 2024-04-19T02:03:30.804 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.804 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.804 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.804 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778846+0000", 2024-04-19T02:03:30.804 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1793999999999996e-05 2024-04-19T02:03:30.804 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.804 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.804 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.804 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.804 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.805 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.es3 42/26 ECSubWrite(tid=32, reqid=osd.7.0:32, at_version=42'25, trim_to=0'0, roll_forward_to=40'23))", 2024-04-19T02:03:30.805 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.755373+0000", 2024-04-19T02:03:30.805 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4.0364458360000004, 2024-04-19T02:03:30.805 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020691993999999998, 2024-04-19T02:03:30.805 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.805 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.805 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.805 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.805 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.805 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.805 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755373+0000", 2024-04-19T02:03:30.805 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.805 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.805 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.806 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.806 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755373+0000", 2024-04-19T02:03:30.806 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.806 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.806 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.806 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.806 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755373+0000", 2024-04-19T02:03:30.806 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5300000000000004e-07 2024-04-19T02:03:30.806 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.806 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.806 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.806 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755450+0000", 2024-04-19T02:03:30.806 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6663999999999996e-05 2024-04-19T02:03:30.806 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.807 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.807 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.807 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755476+0000", 2024-04-19T02:03:30.807 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6395000000000001e-05 2024-04-19T02:03:30.807 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.807 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.807 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.807 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755496+0000", 2024-04-19T02:03:30.807 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0018e-05 2024-04-19T02:03:30.807 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.807 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.807 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.807 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.757829+0000", 2024-04-19T02:03:30.807 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002333052 2024-04-19T02:03:30.808 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.808 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.808 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.808 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.757857+0000", 2024-04-19T02:03:30.808 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7317999999999998e-05 2024-04-19T02:03:30.808 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.808 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.808 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.808 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.775968+0000", 2024-04-19T02:03:30.808 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018111454999999999 2024-04-19T02:03:30.808 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.808 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.808 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.808 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.776065+0000", 2024-04-19T02:03:30.809 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6539000000000005e-05 2024-04-19T02:03:30.809 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.809 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.809 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.809 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.809 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.809 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.ds1 42/26 ECSubWrite(tid=164, reqid=client.4509.0:363, at_version=42'15, trim_to=0'0, roll_forward_to=39'12))", 2024-04-19T02:03:30.809 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.755537+0000", 2024-04-19T02:03:30.809 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4.0362814120000001, 2024-04-19T02:03:30.809 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017310547999999999, 2024-04-19T02:03:30.809 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.809 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.809 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.809 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.809 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.810 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.810 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755537+0000", 2024-04-19T02:03:30.810 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.810 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.810 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.810 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.810 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755537+0000", 2024-04-19T02:03:30.810 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.810 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.810 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.810 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.810 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755539+0000", 2024-04-19T02:03:30.810 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6929999999999999e-06 2024-04-19T02:03:30.810 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.811 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.811 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.811 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755824+0000", 2024-04-19T02:03:30.811 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028525399999999998 2024-04-19T02:03:30.811 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.811 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.811 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.811 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755856+0000", 2024-04-19T02:03:30.811 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2197999999999999e-05 2024-04-19T02:03:30.811 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.811 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.811 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.811 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755879+0000", 2024-04-19T02:03:30.811 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2378000000000002e-05 2024-04-19T02:03:30.811 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.812 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.812 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.812 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.756203+0000", 2024-04-19T02:03:30.812 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00032417500000000001 2024-04-19T02:03:30.812 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.812 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.812 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.812 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.756228+0000", 2024-04-19T02:03:30.812 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5015e-05 2024-04-19T02:03:30.812 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.812 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.812 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.812 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.772747+0000", 2024-04-19T02:03:30.812 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016518739000000001 2024-04-19T02:03:30.813 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.813 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.813 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.813 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.772848+0000", 2024-04-19T02:03:30.813 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000101096 2024-04-19T02:03:30.813 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.813 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.813 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.813 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.813 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.813 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.4s6 42/26 ECSubWrite(tid=45, reqid=osd.1.0:44, at_version=0'0, trim_to=0'0, roll_forward_to=42'32))", 2024-04-19T02:03:30.813 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.758101+0000", 2024-04-19T02:03:30.813 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4.0337171449999998, 2024-04-19T02:03:30.814 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019506200000000001, 2024-04-19T02:03:30.814 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.814 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.814 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.814 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.814 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.814 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.814 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758101+0000", 2024-04-19T02:03:30.814 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.814 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.814 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.814 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.814 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758101+0000", 2024-04-19T02:03:30.814 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.815 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.815 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.815 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.815 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758103+0000", 2024-04-19T02:03:30.815 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4130000000000001e-06 2024-04-19T02:03:30.815 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.815 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.815 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.815 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758129+0000", 2024-04-19T02:03:30.815 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6502000000000001e-05 2024-04-19T02:03:30.815 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.815 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.815 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.815 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758159+0000", 2024-04-19T02:03:30.815 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9408e-05 2024-04-19T02:03:30.816 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.816 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.816 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.816 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758179+0000", 2024-04-19T02:03:30.816 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0520999999999999e-05 2024-04-19T02:03:30.816 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.816 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.816 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.816 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758994+0000", 2024-04-19T02:03:30.816 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00081438699999999997 2024-04-19T02:03:30.816 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.816 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.816 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.816 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759019+0000", 2024-04-19T02:03:30.816 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5731000000000001e-05 2024-04-19T02:03:30.817 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.817 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.817 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.817 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.777537+0000", 2024-04-19T02:03:30.817 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018517927 2024-04-19T02:03:30.817 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.817 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.817 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.817 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.777608+0000", 2024-04-19T02:03:30.817 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0310999999999998e-05 2024-04-19T02:03:30.817 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.817 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.817 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.817 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.817 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.818 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:371 3.as0 3:54128c7a:::smithi05930959-20:head [read 0~1605632,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:30.818 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.760138+0000", 2024-04-19T02:03:30.818 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4.0316807199999998, 2024-04-19T02:03:30.818 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.031633367000000003, 2024-04-19T02:03:30.818 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.818 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.818 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:30.818 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:30.818 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:30.818 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:30.818 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 371 2024-04-19T02:03:30.818 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.818 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.819 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.819 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.819 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760138+0000", 2024-04-19T02:03:30.819 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.819 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.819 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.819 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.819 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760138+0000", 2024-04-19T02:03:30.819 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.819 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.819 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.819 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.819 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760141+0000", 2024-04-19T02:03:30.820 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0790000000000001e-06 2024-04-19T02:03:30.820 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.820 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.820 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.820 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760155+0000", 2024-04-19T02:03:30.820 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4083000000000001e-05 2024-04-19T02:03:30.820 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.820 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.820 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.820 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760187+0000", 2024-04-19T02:03:30.820 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1964000000000001e-05 2024-04-19T02:03:30.820 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.820 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.820 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.820 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760208+0000", 2024-04-19T02:03:30.821 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1395000000000001e-05 2024-04-19T02:03:30.821 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.821 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.821 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.821 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760524+0000", 2024-04-19T02:03:30.821 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00031564199999999998 2024-04-19T02:03:30.821 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.821 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.821 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:30.821 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760793+0000", 2024-04-19T02:03:30.821 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000268819 2024-04-19T02:03:30.821 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.821 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.821 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.822 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774492+0000", 2024-04-19T02:03:30.822 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013699065999999999 2024-04-19T02:03:30.822 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.822 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.822 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:30.822 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774730+0000", 2024-04-19T02:03:30.822 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00023788499999999999 2024-04-19T02:03:30.822 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.822 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.822 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.822 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.791771+0000", 2024-04-19T02:03:30.822 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017041434000000001 2024-04-19T02:03:30.822 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.822 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.823 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.823 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.823 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.823 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:372 3.as0 3:54128c7a:::smithi05930959-20:head [read 0~1605632,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:30.823 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.760260+0000", 2024-04-19T02:03:30.823 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4.0315583530000003, 2024-04-19T02:03:30.823 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.034139714000000002, 2024-04-19T02:03:30.823 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.823 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.823 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:30.823 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:30.823 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:30.823 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:30.823 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 372 2024-04-19T02:03:30.824 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.824 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.824 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.824 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.824 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760260+0000", 2024-04-19T02:03:30.824 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.824 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.824 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.824 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.824 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760260+0000", 2024-04-19T02:03:30.824 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.824 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.824 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.824 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.824 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760262+0000", 2024-04-19T02:03:30.825 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6029999999999999e-06 2024-04-19T02:03:30.825 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.825 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.825 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.825 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760272+0000", 2024-04-19T02:03:30.825 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6560000000000004e-06 2024-04-19T02:03:30.825 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.825 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.825 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.825 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760298+0000", 2024-04-19T02:03:30.825 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6625e-05 2024-04-19T02:03:30.825 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.826 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.826 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.826 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760316+0000", 2024-04-19T02:03:30.826 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7459e-05 2024-04-19T02:03:30.826 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.826 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.826 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.826 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760885+0000", 2024-04-19T02:03:30.826 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00056941200000000004 2024-04-19T02:03:30.826 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.826 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.826 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:30.826 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761126+0000", 2024-04-19T02:03:30.826 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000241124 2024-04-19T02:03:30.827 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.827 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.827 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.827 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.775702+0000", 2024-04-19T02:03:30.827 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014575764 2024-04-19T02:03:30.827 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.827 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.827 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:30.827 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.777674+0000", 2024-04-19T02:03:30.827 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001972297 2024-04-19T02:03:30.827 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.827 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.827 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.828 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.794400+0000", 2024-04-19T02:03:30.828 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016725773999999999 2024-04-19T02:03:30.828 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.828 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.828 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.828 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.828 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.828 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:373 3.as0 3:54128c7a:::smithi05930959-20:head [read 0~1605632,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:30.828 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.760341+0000", 2024-04-19T02:03:30.828 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4.0314778990000004, 2024-04-19T02:03:30.828 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.043665984999999997, 2024-04-19T02:03:30.828 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.828 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.828 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:30.828 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:30.828 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:30.829 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:30.829 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 373 2024-04-19T02:03:30.829 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.829 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.829 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.829 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.829 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760341+0000", 2024-04-19T02:03:30.829 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.829 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.829 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.829 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.829 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760341+0000", 2024-04-19T02:03:30.829 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.829 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.830 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.830 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.830 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760342+0000", 2024-04-19T02:03:30.830 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.497e-06 2024-04-19T02:03:30.830 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.830 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.830 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.830 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760350+0000", 2024-04-19T02:03:30.830 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1119999999999996e-06 2024-04-19T02:03:30.830 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.830 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.830 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.830 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760370+0000", 2024-04-19T02:03:30.830 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9219e-05 2024-04-19T02:03:30.830 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.831 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.831 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.831 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760385+0000", 2024-04-19T02:03:30.831 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5905000000000001e-05 2024-04-19T02:03:30.831 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.831 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.831 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.831 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761214+0000", 2024-04-19T02:03:30.831 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000828939 2024-04-19T02:03:30.831 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.831 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.831 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:30.831 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761439+0000", 2024-04-19T02:03:30.831 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022493900000000001 2024-04-19T02:03:30.832 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.832 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.832 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.832 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.794533+0000", 2024-04-19T02:03:30.832 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.033094077999999999 2024-04-19T02:03:30.832 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.832 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.832 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:30.832 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.794696+0000", 2024-04-19T02:03:30.832 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000162442 2024-04-19T02:03:30.832 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.832 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.832 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.832 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.804007+0000", 2024-04-19T02:03:30.832 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0093108540000000004 2024-04-19T02:03:30.832 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.833 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.833 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.833 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.833 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.833 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.4s6 42/26 ECSubWrite(tid=46, reqid=client.4509.0:374, at_version=42'34, trim_to=0'0, roll_forward_to=42'32))", 2024-04-19T02:03:30.833 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.761536+0000", 2024-04-19T02:03:30.833 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4.0302830350000001, 2024-04-19T02:03:30.833 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016264489, 2024-04-19T02:03:30.833 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.833 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.833 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.833 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.833 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.833 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.833 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761536+0000", 2024-04-19T02:03:30.834 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.834 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.834 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.834 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.834 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761536+0000", 2024-04-19T02:03:30.834 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.834 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.834 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.834 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.834 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761537+0000", 2024-04-19T02:03:30.834 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4959999999999999e-06 2024-04-19T02:03:30.834 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.834 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.834 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.835 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761586+0000", 2024-04-19T02:03:30.835 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9150000000000002e-05 2024-04-19T02:03:30.835 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.835 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.835 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.835 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761614+0000", 2024-04-19T02:03:30.835 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7784e-05 2024-04-19T02:03:30.835 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.835 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.835 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.835 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761648+0000", 2024-04-19T02:03:30.835 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4063999999999998e-05 2024-04-19T02:03:30.835 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.835 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.835 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.836 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761801+0000", 2024-04-19T02:03:30.836 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015288500000000001 2024-04-19T02:03:30.836 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.836 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.836 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.836 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761825+0000", 2024-04-19T02:03:30.836 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4054e-05 2024-04-19T02:03:30.836 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.836 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.836 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.836 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.777679+0000", 2024-04-19T02:03:30.836 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015854322000000001 2024-04-19T02:03:30.836 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.836 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.836 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.836 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.777800+0000", 2024-04-19T02:03:30.837 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000120734 2024-04-19T02:03:30.837 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.837 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.837 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.837 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.837 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.837 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.cs2 42/26 ECSubWrite(tid=32, reqid=osd.9.0:32, at_version=42'18, trim_to=0'0, roll_forward_to=40'16))", 2024-04-19T02:03:30.837 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.766761+0000", 2024-04-19T02:03:30.837 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4.0250577679999999, 2024-04-19T02:03:30.837 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019330330999999999, 2024-04-19T02:03:30.837 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.837 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.837 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.837 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.838 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.838 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.838 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.766761+0000", 2024-04-19T02:03:30.838 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.838 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.838 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.838 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.838 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.766761+0000", 2024-04-19T02:03:30.838 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.838 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.838 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.838 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.838 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.766762+0000", 2024-04-19T02:03:30.838 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9699999999999995e-07 2024-04-19T02:03:30.839 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.839 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.839 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.839 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.766820+0000", 2024-04-19T02:03:30.839 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7865000000000002e-05 2024-04-19T02:03:30.839 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.839 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.839 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.839 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.766843+0000", 2024-04-19T02:03:30.839 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3830999999999999e-05 2024-04-19T02:03:30.839 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.839 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.839 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.839 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.766861+0000", 2024-04-19T02:03:30.839 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7768999999999999e-05 2024-04-19T02:03:30.840 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.840 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.840 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.840 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.767360+0000", 2024-04-19T02:03:30.840 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00049900400000000005 2024-04-19T02:03:30.840 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.840 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.840 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.840 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.767385+0000", 2024-04-19T02:03:30.840 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4808999999999999e-05 2024-04-19T02:03:30.840 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.840 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.840 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.840 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.781492+0000", 2024-04-19T02:03:30.840 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.01410702 2024-04-19T02:03:30.840 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.841 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.841 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.841 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.786091+0000", 2024-04-19T02:03:30.841 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0045992359999999996 2024-04-19T02:03:30.841 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.841 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.841 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.841 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.841 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.841 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/26 ECSubWrite(tid=140, reqid=osd.3.0:134, at_version=0'0, trim_to=0'0, roll_forward_to=42'65))", 2024-04-19T02:03:30.841 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.779396+0000", 2024-04-19T02:03:30.841 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4.0124221369999997, 2024-04-19T02:03:30.841 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020804339000000002, 2024-04-19T02:03:30.841 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.842 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.842 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.842 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.842 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.842 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.842 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.779396+0000", 2024-04-19T02:03:30.842 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.842 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.842 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.842 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.842 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.779396+0000", 2024-04-19T02:03:30.842 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.842 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.842 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.843 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.843 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.779397+0000", 2024-04-19T02:03:30.843 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.9999999999999995e-07 2024-04-19T02:03:30.843 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.843 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.843 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.843 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.779421+0000", 2024-04-19T02:03:30.843 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3226000000000001e-05 2024-04-19T02:03:30.843 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.843 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.843 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.843 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.779468+0000", 2024-04-19T02:03:30.843 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7497000000000003e-05 2024-04-19T02:03:30.843 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.843 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.844 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.844 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.779487+0000", 2024-04-19T02:03:30.844 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9194999999999998e-05 2024-04-19T02:03:30.844 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.844 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.844 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.844 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.786927+0000", 2024-04-19T02:03:30.844 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0074392779999999997 2024-04-19T02:03:30.844 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.844 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.844 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.844 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.786949+0000", 2024-04-19T02:03:30.844 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2572999999999999e-05 2024-04-19T02:03:30.844 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.844 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.844 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.845 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.800145+0000", 2024-04-19T02:03:30.845 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013195475999999999 2024-04-19T02:03:30.845 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.845 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.845 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.845 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.800201+0000", 2024-04-19T02:03:30.845 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6094000000000002e-05 2024-04-19T02:03:30.845 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.845 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.845 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.845 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.845 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.845 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/26 ECSubWrite(tid=141, reqid=osd.3.0:141, at_version=42'67, trim_to=0'0, roll_forward_to=42'65))", 2024-04-19T02:03:30.845 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.781731+0000", 2024-04-19T02:03:30.845 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4.0100873740000003, 2024-04-19T02:03:30.845 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018514591, 2024-04-19T02:03:30.846 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.846 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.846 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.846 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.846 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.846 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.846 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.781731+0000", 2024-04-19T02:03:30.846 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.846 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.846 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.846 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.846 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.781731+0000", 2024-04-19T02:03:30.846 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.847 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.847 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.847 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.847 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.781733+0000", 2024-04-19T02:03:30.847 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.471e-06 2024-04-19T02:03:30.847 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.847 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.847 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.847 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.781778+0000", 2024-04-19T02:03:30.847 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5661999999999998e-05 2024-04-19T02:03:30.847 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.847 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.847 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.847 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.786046+0000", 2024-04-19T02:03:30.848 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0042675769999999998 2024-04-19T02:03:30.848 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.848 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.848 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.848 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.786078+0000", 2024-04-19T02:03:30.848 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2088999999999997e-05 2024-04-19T02:03:30.848 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.848 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.848 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.848 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.787725+0000", 2024-04-19T02:03:30.848 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001646972 2024-04-19T02:03:30.848 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.848 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.848 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.848 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.787746+0000", 2024-04-19T02:03:30.849 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0809999999999999e-05 2024-04-19T02:03:30.849 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.849 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.849 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.849 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.800210+0000", 2024-04-19T02:03:30.849 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012463933999999999 2024-04-19T02:03:30.849 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.849 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.849 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.849 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.800246+0000", 2024-04-19T02:03:30.849 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6075999999999998e-05 2024-04-19T02:03:30.849 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.849 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.849 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.849 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.850 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.850 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:381 3.as0 3:512144ec:::smithi05930959-48:head [create,setxattr pnc]aaY (11788) in=7b,setxattr ZTVVStUpk__YaS\\blR\\XgesYT[Or[[aUett_nil] (1859) in=40b,setxattr ZtotWOVcPP_oiTUUTlqtYZsNneQnmhc\\bNdNpsgo (1402) in=40b,setxattr cR\\akUitgYfao[oisZ^ach\\i_kdge\\[c_mbmUf]^ (313) in=40b,setxattr pRe[UcWhXR^oagaSYScYPQa]r[hdldO\\S`V^[nh^ (874) in=40b,setxattr srVXOoW^mbi\\ccUpc`barbsc (11788) in=24b,setxattr pqjhUkafUjRqdpObgi_ZX^QV (11788) in=24b,setxattr q]Of\\bd_VgdoaP\\k^tmmpUqq^otrVVOdrtgU]nVk (12280) in=40b,setxattr ]Ple__\\nrXPQYecmQXf_o^bgN`mofssSPQrYslSO (1859) in=40b,setxattr ZiQcjT`UZQ]es__dmo^h]NdV^`qT_hniQtVY`WsT (770) in=40b,setxattr gpT^oWmeYZpbVdojjWnUWO[YfZptiQhdWdhs]Zo^ (841) in=40b,setxattr `Zgeqf\\Rb][aUdTVXYhj[ealjcUnRaiY (11788) in=32b,setxattr [dN]aooYVPYS^Q`_`bqSNVSS^`ZT\\csYZSrnjtUT (11296) in=40b,setxattr ehcNktRfRjqWWi[^XWcZokN]Wb[YnkeNl\\lkt_t\\ (1262) in=40b,setxattr OdsZcnit[de]qNV (11788) in=15b,setxattr ^]^lenZo\\TjoZOpONRno]jtaWOd`tqNdWcYccWNk (560) in=40b,setxattr ZUohXUqoPmYaaTceWo\\SRigth\\qP[Tg_^^asjknX (1085) in=40b,setxattr lR`dUarlc[n`ctj^TlY\\QpagN_doPfV`lZQ^kNtY (526) in=40b,setxattr [iXNTskqfmN\\qVhqrm`rdXpaN_VaeRbd^O\\Z_gcY (1261) in=40b,setxattr t_jR_akNV_aXVR^icaga]cP]baNne_f_UdsoeVUk (1894) in=40b] snapc 9=[8,6] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T02:03:30.850 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:27.087092+0000", 2024-04-19T02:03:30.850 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 3.7047269050000002, 2024-04-19T02:03:30.850 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.025861967999999999, 2024-04-19T02:03:30.850 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.850 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.850 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:30.850 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:30.850 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:30.850 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:30.850 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 381 2024-04-19T02:03:30.850 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.850 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.850 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.851 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.851 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087092+0000", 2024-04-19T02:03:30.851 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.851 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.851 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.851 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.851 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087092+0000", 2024-04-19T02:03:30.851 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.851 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.851 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.851 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.851 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087095+0000", 2024-04-19T02:03:30.851 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2799999999999999e-06 2024-04-19T02:03:30.851 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.852 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.852 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.852 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087306+0000", 2024-04-19T02:03:30.852 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021136000000000001 2024-04-19T02:03:30.852 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.852 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.852 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.852 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087341+0000", 2024-04-19T02:03:30.852 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4654999999999999e-05 2024-04-19T02:03:30.852 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.852 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.852 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.852 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087358+0000", 2024-04-19T02:03:30.852 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7261e-05 2024-04-19T02:03:30.853 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.853 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.853 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.853 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087536+0000", 2024-04-19T02:03:30.853 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000177424 2024-04-19T02:03:30.853 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.853 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.853 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op must wait for map", 2024-04-19T02:03:30.853 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087551+0000", 2024-04-19T02:03:30.853 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5362e-05 2024-04-19T02:03:30.853 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.853 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.853 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.853 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.092762+0000", 2024-04-19T02:03:30.854 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0052113589999999996 2024-04-19T02:03:30.854 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.854 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.854 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:30.854 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.092952+0000", 2024-04-19T02:03:30.854 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00018992100000000001 2024-04-19T02:03:30.854 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.854 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.854 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.854 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.095255+0000", 2024-04-19T02:03:30.854 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0023027030000000001 2024-04-19T02:03:30.854 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.854 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.855 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.855 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.101758+0000", 2024-04-19T02:03:30.855 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0065034380000000003 2024-04-19T02:03:30.855 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.855 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.855 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:30.855 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.112669+0000", 2024-04-19T02:03:30.855 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010910737 2024-04-19T02:03:30.855 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.855 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.855 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.855 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.112954+0000", 2024-04-19T02:03:30.855 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028446800000000001 2024-04-19T02:03:30.855 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.855 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.855 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.856 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.856 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.856 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:434 3.as0 3:5a72bad2:::smithi05930959-41:head [read 0~3293184,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:30.856 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:30.194592+0000", 2024-04-19T02:03:30.856 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.59722694600000004, 2024-04-19T02:03:30.856 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020994932000000001, 2024-04-19T02:03:30.856 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.856 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.856 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:30.856 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:30.856 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:30.856 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:30.856 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 434 2024-04-19T02:03:30.856 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.857 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.857 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.857 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.857 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194592+0000", 2024-04-19T02:03:30.857 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.857 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.857 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.857 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.857 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194592+0000", 2024-04-19T02:03:30.857 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.857 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.857 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.857 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.857 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194594+0000", 2024-04-19T02:03:30.857 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6730000000000001e-06 2024-04-19T02:03:30.858 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.858 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.858 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.858 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194608+0000", 2024-04-19T02:03:30.858 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3382e-05 2024-04-19T02:03:30.858 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.858 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.858 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.858 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194638+0000", 2024-04-19T02:03:30.858 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0691000000000001e-05 2024-04-19T02:03:30.858 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.858 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.859 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.859 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194659+0000", 2024-04-19T02:03:30.859 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0182000000000001e-05 2024-04-19T02:03:30.859 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.859 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.859 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.859 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194837+0000", 2024-04-19T02:03:30.859 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017831800000000001 2024-04-19T02:03:30.859 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.859 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.859 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:30.859 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194983+0000", 2024-04-19T02:03:30.859 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014610299999999999 2024-04-19T02:03:30.859 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.860 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.860 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.860 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.215587+0000", 2024-04-19T02:03:30.860 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020603583000000002 2024-04-19T02:03:30.860 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.860 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.860 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.860 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.860 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.860 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:435 3.as0 3:5a72bad2:::smithi05930959-41:head [read 0~3293184,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:30.860 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:30.194714+0000", 2024-04-19T02:03:30.860 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.59710506500000005, 2024-04-19T02:03:30.860 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.025824666, 2024-04-19T02:03:30.860 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.860 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.861 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:30.861 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:30.861 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:30.861 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:30.861 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 435 2024-04-19T02:03:30.861 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.861 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.861 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.861 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.861 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194714+0000", 2024-04-19T02:03:30.861 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.861 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.861 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.861 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.861 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194714+0000", 2024-04-19T02:03:30.862 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.862 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.862 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.862 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.862 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194715+0000", 2024-04-19T02:03:30.862 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.497e-06 2024-04-19T02:03:30.862 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.862 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.862 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.862 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194727+0000", 2024-04-19T02:03:30.862 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2021999999999999e-05 2024-04-19T02:03:30.862 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.862 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.863 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.863 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194772+0000", 2024-04-19T02:03:30.863 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4620000000000003e-05 2024-04-19T02:03:30.863 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.863 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.863 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.863 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194790+0000", 2024-04-19T02:03:30.863 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7846000000000002e-05 2024-04-19T02:03:30.863 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.863 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.863 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.863 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.195633+0000", 2024-04-19T02:03:30.863 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00084348400000000003 2024-04-19T02:03:30.863 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.863 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.864 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:30.864 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.195885+0000", 2024-04-19T02:03:30.864 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00025215900000000001 2024-04-19T02:03:30.864 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.864 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.864 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.864 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.220538+0000", 2024-04-19T02:03:30.864 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.024653037999999999 2024-04-19T02:03:30.864 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.864 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.864 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.864 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.864 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.864 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:436 3.as0 3:5a72bad2:::smithi05930959-41:head [read 0~3293184,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:30.865 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:30.194819+0000", 2024-04-19T02:03:30.865 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.59699960200000002, 2024-04-19T02:03:30.865 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.028970988999999999, 2024-04-19T02:03:30.865 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.865 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.865 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:30.865 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:30.865 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:30.865 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:30.865 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 436 2024-04-19T02:03:30.865 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.865 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.865 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.865 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.865 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194819+0000", 2024-04-19T02:03:30.866 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.866 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.866 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.866 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.866 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194819+0000", 2024-04-19T02:03:30.866 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.866 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.866 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.866 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.866 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194820+0000", 2024-04-19T02:03:30.866 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1829999999999999e-06 2024-04-19T02:03:30.866 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.866 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.866 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.867 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194828+0000", 2024-04-19T02:03:30.867 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.306e-06 2024-04-19T02:03:30.867 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.867 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.867 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.867 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194846+0000", 2024-04-19T02:03:30.867 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8592000000000001e-05 2024-04-19T02:03:30.867 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.867 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.867 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.867 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194860+0000", 2024-04-19T02:03:30.867 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4185999999999999e-05 2024-04-19T02:03:30.867 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.867 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.867 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.868 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198124+0000", 2024-04-19T02:03:30.868 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0032639430000000001 2024-04-19T02:03:30.868 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.868 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.868 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:30.868 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198217+0000", 2024-04-19T02:03:30.868 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2303e-05 2024-04-19T02:03:30.868 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.868 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.868 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.868 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.223790+0000", 2024-04-19T02:03:30.868 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.025573476000000001 2024-04-19T02:03:30.868 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.868 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.869 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.869 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.869 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.869 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s6 46/26 ECSubWrite(tid=162, reqid=client.4509.0:424, at_version=46'77, trim_to=0'0, roll_forward_to=46'75))", 2024-04-19T02:03:30.869 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:30.196958+0000", 2024-04-19T02:03:30.869 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.59486091799999996, 2024-04-19T02:03:30.869 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018334775000000001, 2024-04-19T02:03:30.869 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.869 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.869 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.869 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.869 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.869 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.869 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.196958+0000", 2024-04-19T02:03:30.869 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.870 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.870 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.870 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.870 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.196958+0000", 2024-04-19T02:03:30.870 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.870 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.870 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.870 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.870 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.196960+0000", 2024-04-19T02:03:30.870 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9759999999999998e-06 2024-04-19T02:03:30.870 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.870 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.870 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.870 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.197014+0000", 2024-04-19T02:03:30.870 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4589999999999997e-05 2024-04-19T02:03:30.871 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.871 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.871 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.871 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.197047+0000", 2024-04-19T02:03:30.871 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2727e-05 2024-04-19T02:03:30.871 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.871 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.871 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.871 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.197070+0000", 2024-04-19T02:03:30.871 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2823000000000002e-05 2024-04-19T02:03:30.871 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.871 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.871 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.871 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.197364+0000", 2024-04-19T02:03:30.871 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000293725 2024-04-19T02:03:30.872 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.872 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.872 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.872 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.197389+0000", 2024-04-19T02:03:30.872 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5422000000000002e-05 2024-04-19T02:03:30.872 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.872 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.872 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.872 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.215208+0000", 2024-04-19T02:03:30.872 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017819360999999999 2024-04-19T02:03:30.872 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.872 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.872 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.872 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.215292+0000", 2024-04-19T02:03:30.872 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4151000000000001e-05 2024-04-19T02:03:30.872 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.873 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.873 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.873 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.873 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.873 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s6 46/26 ECSubWrite(tid=163, reqid=client.4509.0:425, at_version=46'78, trim_to=0'0, roll_forward_to=46'75))", 2024-04-19T02:03:30.873 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:30.198808+0000", 2024-04-19T02:03:30.873 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.59301079400000001, 2024-04-19T02:03:30.873 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021976591, 2024-04-19T02:03:30.873 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.873 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.873 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.873 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.873 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.873 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.873 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198808+0000", 2024-04-19T02:03:30.873 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.873 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.873 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.874 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.874 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198808+0000", 2024-04-19T02:03:30.874 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.874 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.874 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.874 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.874 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198809+0000", 2024-04-19T02:03:30.874 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6330000000000001e-06 2024-04-19T02:03:30.874 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.874 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.874 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.874 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198856+0000", 2024-04-19T02:03:30.874 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6414e-05 2024-04-19T02:03:30.874 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.875 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.875 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.875 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198882+0000", 2024-04-19T02:03:30.875 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6523999999999999e-05 2024-04-19T02:03:30.875 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.875 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.875 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.875 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198901+0000", 2024-04-19T02:03:30.875 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8763e-05 2024-04-19T02:03:30.875 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.875 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.875 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.875 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.202201+0000", 2024-04-19T02:03:30.875 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0033002190000000001 2024-04-19T02:03:30.875 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.876 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.876 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.876 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.202210+0000", 2024-04-19T02:03:30.876 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7830000000000004e-06 2024-04-19T02:03:30.876 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.876 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.876 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.876 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.220681+0000", 2024-04-19T02:03:30.876 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018470594999999999 2024-04-19T02:03:30.876 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.876 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.876 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.876 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.220784+0000", 2024-04-19T02:03:30.876 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010365999999999999 2024-04-19T02:03:30.876 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.876 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.877 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.877 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.877 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.877 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s6 46/26 ECSubWrite(tid=164, reqid=client.4509.0:426, at_version=46'79, trim_to=0'0, roll_forward_to=46'75))", 2024-04-19T02:03:30.877 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:30.201234+0000", 2024-04-19T02:03:30.877 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.59058482300000004, 2024-04-19T02:03:30.877 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019620385000000001, 2024-04-19T02:03:30.877 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:30.877 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:30.877 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:30.877 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:30.877 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.877 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:30.877 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.201234+0000", 2024-04-19T02:03:30.877 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.878 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.878 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.878 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:30.878 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.201234+0000", 2024-04-19T02:03:30.878 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:30.878 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.878 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.878 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:30.878 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.201235+0000", 2024-04-19T02:03:30.878 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.686e-06 2024-04-19T02:03:30.878 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.878 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.878 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:30.878 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.201294+0000", 2024-04-19T02:03:30.878 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8936e-05 2024-04-19T02:03:30.879 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.879 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.879 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:30.879 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.201327+0000", 2024-04-19T02:03:30.879 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2715000000000001e-05 2024-04-19T02:03:30.879 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.879 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.879 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:30.879 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.201350+0000", 2024-04-19T02:03:30.879 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2381000000000001e-05 2024-04-19T02:03:30.879 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.879 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.879 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:30.879 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.205516+0000", 2024-04-19T02:03:30.879 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0041668199999999999 2024-04-19T02:03:30.879 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.879 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.880 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:30.880 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.205748+0000", 2024-04-19T02:03:30.880 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000231654 2024-04-19T02:03:30.880 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.880 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.880 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:30.880 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.220792+0000", 2024-04-19T02:03:30.880 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015044014 2024-04-19T02:03:30.880 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:30.880 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:30.880 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:30.880 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.220854+0000", 2024-04-19T02:03:30.880 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2179000000000002e-05 2024-04-19T02:03:30.880 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.880 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.881 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.881 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:30.881 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:30.881 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:30.882 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-19T02:03:30.883 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:30.882+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:30.983 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:03:30.986+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:03:31.016 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:31.016 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:03:31.016 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-04-19T02:03:31.017 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:31.031 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-19T02:03:31.075 INFO:tasks.rados.rados.0.smithi059.stdout:172: done (2 left) 2024-04-19T02:03:31.076 INFO:tasks.rados.rados.0.smithi059.stdout:173: rollback oid 2 current snap is 9 2024-04-19T02:03:31.076 INFO:tasks.rados.rados.0.smithi059.stdout:173: done (2 left) 2024-04-19T02:03:31.076 INFO:tasks.rados.rados.0.smithi059.stdout:174: setattr oid 27 current snap is 9 2024-04-19T02:03:31.076 INFO:tasks.rados.rados.0.smithi059.stdout:170: finishing write tid 2 to smithi05930959-7 2024-04-19T02:03:31.076 INFO:tasks.rados.rados.0.smithi059.stdout:170: finishing write tid 3 to smithi05930959-7 2024-04-19T02:03:31.076 INFO:tasks.rados.rados.0.smithi059.stdout:170: finishing write tid 4 to smithi05930959-7 2024-04-19T02:03:31.076 INFO:tasks.rados.rados.0.smithi059.stdout:171: expect (ObjNum 52 snap 0 seq_num 52) 2024-04-19T02:03:31.078 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.4 to 1 (10000) 2024-04-19T02:03:31.085 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:31.082+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:31.109 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-04-19T02:03:31.134 INFO:tasks.rados.rados.0.smithi059.stdout:170: finishing write tid 5 to smithi05930959-7 2024-04-19T02:03:31.134 INFO:tasks.rados.rados.0.smithi059.stdout:170: oid 7 updating version 0 to 33 2024-04-19T02:03:31.134 INFO:tasks.rados.rados.0.smithi059.stdout:170: oid 7 version 33 is already newer than 32 2024-04-19T02:03:31.134 INFO:tasks.rados.rados.0.smithi059.stdout:170: oid 7 version 33 is already newer than 30 2024-04-19T02:03:31.134 INFO:tasks.rados.rados.0.smithi059.stdout:170: oid 7 version 33 is already newer than 31 2024-04-19T02:03:31.134 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 7 v 33 (ObjNum 91 snap 9 seq_num 91) dirty exists 2024-04-19T02:03:31.134 INFO:tasks.rados.rados.0.smithi059.stdout:170: left oid 7 (ObjNum 91 snap 9 seq_num 91) 2024-04-19T02:03:31.134 INFO:tasks.rados.rados.0.smithi059.stdout:170: done (2 left) 2024-04-19T02:03:31.134 INFO:tasks.rados.rados.0.smithi059.stdout:171: done (1 left) 2024-04-19T02:03:31.135 INFO:tasks.rados.rados.0.smithi059.stdout:175: rollback oid 49 current snap is 9 2024-04-19T02:03:31.135 INFO:tasks.rados.rados.0.smithi059.stdout:175: done (1 left) 2024-04-19T02:03:31.135 INFO:tasks.rados.rados.0.smithi059.stdout:176: read oid 34 snap -1 2024-04-19T02:03:31.135 INFO:tasks.rados.rados.0.smithi059.stdout:176: expect (ObjNum 82 snap 7 seq_num 82) 2024-04-19T02:03:31.135 INFO:tasks.rados.rados.0.smithi059.stdout:177: read oid 46 snap -1 2024-04-19T02:03:31.135 INFO:tasks.rados.rados.0.smithi059.stdout:177: expect (ObjNum 45 snap 0 seq_num 45) 2024-04-19T02:03:31.135 INFO:tasks.rados.rados.0.smithi059.stdout:178: setattr oid 16 current snap is 9 2024-04-19T02:03:31.136 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 36 current snap is 9 2024-04-19T02:03:31.136 INFO:tasks.rados.rados.0.smithi059.stdout:179: seq_num 92 ranges {2080768=786432,2867200=491520,3358720=278528} 2024-04-19T02:03:31.152 INFO:tasks.rados.rados.0.smithi059.stdout:179: writing smithi05930959-36 from 2080768 to 2867200 tid 1 2024-04-19T02:03:31.155 INFO:tasks.rados.rados.0.smithi059.stdout:179: writing smithi05930959-36 from 2867200 to 3358720 tid 2 2024-04-19T02:03:31.158 INFO:tasks.rados.rados.0.smithi059.stdout:179: writing smithi05930959-36 from 3358720 to 3637248 tid 3 2024-04-19T02:03:31.158 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 45 current snap is 9 2024-04-19T02:03:31.158 INFO:tasks.rados.rados.0.smithi059.stdout:180: seq_num 93 ranges {0=802816,802816=638976,1441792=131072} 2024-04-19T02:03:31.163 INFO:tasks.rados.rados.0.smithi059.stdout:180: writing smithi05930959-45 from 0 to 802816 tid 1 2024-04-19T02:03:31.167 INFO:tasks.rados.rados.0.smithi059.stdout:180: writing smithi05930959-45 from 802816 to 1441792 tid 2 2024-04-19T02:03:31.168 INFO:tasks.rados.rados.0.smithi059.stdout:180: writing smithi05930959-45 from 1441792 to 1572864 tid 3 2024-04-19T02:03:31.169 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 20 current snap is 9 2024-04-19T02:03:31.169 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 27 v 92 (ObjNum 79 snap 6 seq_num 79) dirty exists 2024-04-19T02:03:31.169 INFO:tasks.rados.rados.0.smithi059.stdout:181: seq_num 94 ranges {1605632=425984,2031616=786432,2818048=376832} 2024-04-19T02:03:31.180 INFO:tasks.rados.rados.0.smithi059.stdout:181: writing smithi05930959-20 from 1605632 to 2031616 tid 1 2024-04-19T02:03:31.185 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:31.182+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:31.185 INFO:tasks.rados.rados.0.smithi059.stdout:181: writing smithi05930959-20 from 2031616 to 2818048 tid 2 2024-04-19T02:03:31.187 INFO:tasks.rados.rados.0.smithi059.stdout:181: writing smithi05930959-20 from 2818048 to 3194880 tid 3 2024-04-19T02:03:31.188 INFO:tasks.rados.rados.0.smithi059.stdout:174: done (6 left) 2024-04-19T02:03:31.188 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 39 current snap is 9 2024-04-19T02:03:31.188 INFO:tasks.rados.rados.0.smithi059.stdout:182: seq_num 95 ranges {1966080=442368,2408448=540672,2949120=622592} 2024-04-19T02:03:31.195 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:31.195 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:03:31.195 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-04-19T02:03:31.195 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-04-19T02:03:31.195 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:31.202 INFO:tasks.rados.rados.0.smithi059.stdout:182: writing smithi05930959-39 from 1966080 to 2408448 tid 1 2024-04-19T02:03:31.205 INFO:tasks.rados.rados.0.smithi059.stdout:182: writing smithi05930959-39 from 2408448 to 2949120 tid 2 2024-04-19T02:03:31.209 INFO:tasks.rados.rados.0.smithi059.stdout:182: writing smithi05930959-39 from 2949120 to 3571712 tid 3 2024-04-19T02:03:31.210 INFO:tasks.rados.rados.0.smithi059.stdout:183: read oid 17 snap -1 2024-04-19T02:03:31.210 INFO:tasks.rados.rados.0.smithi059.stdout:183: expect (ObjNum 74 snap 6 seq_num 74) 2024-04-19T02:03:31.210 INFO:tasks.rados.rados.0.smithi059.stdout:184: rmattr oid 3 current snap is 9 2024-04-19T02:03:31.210 INFO:tasks.rados.rados.0.smithi059.stdout:184: done (8 left) 2024-04-19T02:03:31.210 INFO:tasks.rados.rados.0.smithi059.stdout:185: read oid 43 snap -1 2024-04-19T02:03:31.210 INFO:tasks.rados.rados.0.smithi059.stdout:185: expect (ObjNum 74 snap 6 seq_num 74) 2024-04-19T02:03:31.210 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 19 current snap is 9 2024-04-19T02:03:31.210 INFO:tasks.rados.rados.0.smithi059.stdout:186: seq_num 96 ranges {311296=458752,770048=688128,1458176=475136} 2024-04-19T02:03:31.214 INFO:tasks.rados.rados.0.smithi059.stdout:186: writing smithi05930959-19 from 311296 to 770048 tid 1 2024-04-19T02:03:31.219 INFO:tasks.rados.rados.0.smithi059.stdout:186: writing smithi05930959-19 from 770048 to 1458176 tid 2 2024-04-19T02:03:31.221 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-19T02:03:31.223 INFO:tasks.rados.rados.0.smithi059.stdout:186: writing smithi05930959-19 from 1458176 to 1933312 tid 3 2024-04-19T02:03:31.223 INFO:tasks.rados.rados.0.smithi059.stdout:187: rollback oid 14 current snap is 9 2024-04-19T02:03:31.224 INFO:tasks.rados.rados.0.smithi059.stdout:187: done (10 left) 2024-04-19T02:03:31.224 INFO:tasks.rados.rados.0.smithi059.stdout:188: rmattr oid 50 current snap is 9 2024-04-19T02:03:31.224 INFO:tasks.rados.rados.0.smithi059.stdout:188: done (10 left) 2024-04-19T02:03:31.224 INFO:tasks.rados.rados.0.smithi059.stdout:189: snap_create 2024-04-19T02:03:31.224 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 16 v 41 (ObjNum 84 snap 8 seq_num 84) dirty exists 2024-04-19T02:03:31.224 INFO:tasks.rados.rados.0.smithi059.stdout:177: expect (ObjNum 45 snap 0 seq_num 45) 2024-04-19T02:03:31.248 INFO:tasks.rados.rados.0.smithi059.stdout:176: expect (ObjNum 82 snap 7 seq_num 82) 2024-04-19T02:03:31.286 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:03:31.286+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:03:31.308 INFO:tasks.rados.rados.0.smithi059.stdout:179: finishing write tid 1 to smithi05930959-36 2024-04-19T02:03:31.308 INFO:tasks.rados.rados.0.smithi059.stdout:179: finishing write tid 2 to smithi05930959-36 2024-04-19T02:03:31.308 INFO:tasks.rados.rados.0.smithi059.stdout:179: finishing write tid 3 to smithi05930959-36 2024-04-19T02:03:31.308 INFO:tasks.rados.rados.0.smithi059.stdout:179: finishing write tid 4 to smithi05930959-36 2024-04-19T02:03:31.308 INFO:tasks.rados.rados.0.smithi059.stdout:179: finishing write tid 5 to smithi05930959-36 2024-04-19T02:03:31.308 INFO:tasks.rados.rados.0.smithi059.stdout:179: oid 36 updating version 0 to 18 2024-04-19T02:03:31.308 INFO:tasks.rados.rados.0.smithi059.stdout:179: oid 36 updating version 18 to 21 2024-04-19T02:03:31.308 INFO:tasks.rados.rados.0.smithi059.stdout:179: oid 36 version 21 is already newer than 20 2024-04-19T02:03:31.309 INFO:tasks.rados.rados.0.smithi059.stdout:179: oid 36 version 21 is already newer than 19 2024-04-19T02:03:31.309 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 36 v 21 (ObjNum 92 snap 9 seq_num 92) dirty exists 2024-04-19T02:03:31.309 INFO:tasks.rados.rados.0.smithi059.stdout:179: left oid 36 (ObjNum 92 snap 9 seq_num 92) 2024-04-19T02:03:31.309 INFO:tasks.rados.rados.0.smithi059.stdout:180: finishing write tid 1 to smithi05930959-45 2024-04-19T02:03:31.309 INFO:tasks.rados.rados.0.smithi059.stdout:180: finishing write tid 2 to smithi05930959-45 2024-04-19T02:03:31.309 INFO:tasks.rados.rados.0.smithi059.stdout:180: finishing write tid 3 to smithi05930959-45 2024-04-19T02:03:31.309 INFO:tasks.rados.rados.0.smithi059.stdout:180: finishing write tid 4 to smithi05930959-45 2024-04-19T02:03:31.309 INFO:tasks.rados.rados.0.smithi059.stdout:180: finishing write tid 5 to smithi05930959-45 2024-04-19T02:03:31.309 INFO:tasks.rados.rados.0.smithi059.stdout:180: oid 45 updating version 0 to 21 2024-04-19T02:03:31.309 INFO:tasks.rados.rados.0.smithi059.stdout:180: oid 45 updating version 21 to 23 2024-04-19T02:03:31.310 INFO:tasks.rados.rados.0.smithi059.stdout:180: oid 45 version 23 is already newer than 22 2024-04-19T02:03:31.310 INFO:tasks.rados.rados.0.smithi059.stdout:180: oid 45 version 23 is already newer than 20 2024-04-19T02:03:31.310 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 45 v 23 (ObjNum 93 snap 9 seq_num 93) dirty exists 2024-04-19T02:03:31.310 INFO:tasks.rados.rados.0.smithi059.stdout:180: left oid 45 (ObjNum 93 snap 9 seq_num 93) 2024-04-19T02:03:31.310 INFO:tasks.rados.rados.0.smithi059.stdout:181: finishing write tid 1 to smithi05930959-20 2024-04-19T02:03:31.310 INFO:tasks.rados.rados.0.smithi059.stdout:181: finishing write tid 2 to smithi05930959-20 2024-04-19T02:03:31.310 INFO:tasks.rados.rados.0.smithi059.stdout:181: finishing write tid 3 to smithi05930959-20 2024-04-19T02:03:31.310 INFO:tasks.rados.rados.0.smithi059.stdout:181: finishing write tid 4 to smithi05930959-20 2024-04-19T02:03:31.310 INFO:tasks.rados.rados.0.smithi059.stdout:181: finishing write tid 5 to smithi05930959-20 2024-04-19T02:03:31.310 INFO:tasks.rados.rados.0.smithi059.stdout:181: oid 20 updating version 0 to 52 2024-04-19T02:03:31.310 INFO:tasks.rados.rados.0.smithi059.stdout:181: oid 20 updating version 52 to 53 2024-04-19T02:03:31.310 INFO:tasks.rados.rados.0.smithi059.stdout:181: oid 20 updating version 53 to 55 2024-04-19T02:03:31.310 INFO:tasks.rados.rados.0.smithi059.stdout:181: oid 20 version 55 is already newer than 54 2024-04-19T02:03:31.310 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 20 v 55 (ObjNum 94 snap 9 seq_num 94) dirty exists 2024-04-19T02:03:31.311 INFO:tasks.rados.rados.0.smithi059.stdout:181: left oid 20 (ObjNum 94 snap 9 seq_num 94) 2024-04-19T02:03:31.311 INFO:tasks.rados.rados.0.smithi059.stdout:182: finishing write tid 1 to smithi05930959-39 2024-04-19T02:03:31.311 INFO:tasks.rados.rados.0.smithi059.stdout:182: finishing write tid 2 to smithi05930959-39 2024-04-19T02:03:31.311 INFO:tasks.rados.rados.0.smithi059.stdout:182: finishing write tid 3 to smithi05930959-39 2024-04-19T02:03:31.311 INFO:tasks.rados.rados.0.smithi059.stdout:182: finishing write tid 4 to smithi05930959-39 2024-04-19T02:03:31.311 INFO:tasks.rados.rados.0.smithi059.stdout:182: finishing write tid 5 to smithi05930959-39 2024-04-19T02:03:31.311 INFO:tasks.rados.rados.0.smithi059.stdout:182: oid 39 updating version 0 to 48 2024-04-19T02:03:31.311 INFO:tasks.rados.rados.0.smithi059.stdout:182: oid 39 version 48 is already newer than 47 2024-04-19T02:03:31.311 INFO:tasks.rados.rados.0.smithi059.stdout:182: oid 39 version 48 is already newer than 46 2024-04-19T02:03:31.311 INFO:tasks.rados.rados.0.smithi059.stdout:182: oid 39 version 48 is already newer than 45 2024-04-19T02:03:31.311 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 39 v 48 (ObjNum 95 snap 9 seq_num 95) dirty exists 2024-04-19T02:03:31.311 INFO:tasks.rados.rados.0.smithi059.stdout:182: left oid 39 (ObjNum 95 snap 9 seq_num 95) 2024-04-19T02:03:31.311 INFO:tasks.rados.rados.0.smithi059.stdout:185: expect (ObjNum 74 snap 6 seq_num 74) 2024-04-19T02:03:31.342 INFO:tasks.rados.rados.0.smithi059.stdout:183: expect (ObjNum 74 snap 6 seq_num 74) 2024-04-19T02:03:31.343 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:31.343 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-04-19T02:03:31.344 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-04-19T02:03:31.344 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-04-19T02:03:31.344 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.344 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es4 35/26 ECSubWrite(tid=23, reqid=osd.7.0:23, at_version=35'18, trim_to=0'0, roll_forward_to=34'16))", 2024-04-19T02:03:31.344 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.676071+0000", 2024-04-19T02:03:31.344 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 11.668587343, 2024-04-19T02:03:31.344 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0084307070000000008, 2024-04-19T02:03:31.344 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:31.344 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:31.344 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:31.344 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:31.344 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.344 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:31.344 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.676071+0000", 2024-04-19T02:03:31.344 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.345 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.345 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.345 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:31.345 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.676071+0000", 2024-04-19T02:03:31.345 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.345 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.345 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.345 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:31.345 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.676072+0000", 2024-04-19T02:03:31.345 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7000000000000001e-07 2024-04-19T02:03:31.345 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.345 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.345 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:31.345 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.676098+0000", 2024-04-19T02:03:31.345 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6169999999999998e-05 2024-04-19T02:03:31.346 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.346 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.346 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:31.346 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.676110+0000", 2024-04-19T02:03:31.346 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2568000000000001e-05 2024-04-19T02:03:31.346 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.346 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.346 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:31.346 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.676122+0000", 2024-04-19T02:03:31.346 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1371e-05 2024-04-19T02:03:31.346 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.346 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.346 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:31.346 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.676504+0000", 2024-04-19T02:03:31.347 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00038213599999999999 2024-04-19T02:03:31.347 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.347 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.347 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:31.347 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.676513+0000", 2024-04-19T02:03:31.347 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.9509999999999995e-06 2024-04-19T02:03:31.347 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.347 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.347 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:31.347 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.684458+0000", 2024-04-19T02:03:31.347 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0079447419999999994 2024-04-19T02:03:31.347 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.347 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.347 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:31.347 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.684502+0000", 2024-04-19T02:03:31.348 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4499000000000002e-05 2024-04-19T02:03:31.348 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.348 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:31.348 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.348 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.348 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.348 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s7 35/26 ECSubWrite(tid=30, reqid=osd.1.0:29, at_version=0'0, trim_to=0'0, roll_forward_to=35'18))", 2024-04-19T02:03:31.348 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.680702+0000", 2024-04-19T02:03:31.348 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 11.663956689000001, 2024-04-19T02:03:31.348 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0072111470000000002, 2024-04-19T02:03:31.348 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:31.348 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:31.348 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:31.348 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:31.349 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.349 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:31.349 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680702+0000", 2024-04-19T02:03:31.349 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.349 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.349 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.349 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:31.349 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680702+0000", 2024-04-19T02:03:31.349 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.349 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.349 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.349 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:31.349 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680703+0000", 2024-04-19T02:03:31.349 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3099999999999996e-07 2024-04-19T02:03:31.350 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.350 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.350 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:31.350 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680717+0000", 2024-04-19T02:03:31.350 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4515000000000001e-05 2024-04-19T02:03:31.350 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.350 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.350 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:31.350 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680737+0000", 2024-04-19T02:03:31.350 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.984e-05 2024-04-19T02:03:31.350 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.350 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.350 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:31.350 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680747+0000", 2024-04-19T02:03:31.351 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.7410000000000004e-06 2024-04-19T02:03:31.351 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.351 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.351 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:31.351 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680846+0000", 2024-04-19T02:03:31.351 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.8572000000000004e-05 2024-04-19T02:03:31.351 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.351 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.351 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:31.351 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680857+0000", 2024-04-19T02:03:31.351 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1055e-05 2024-04-19T02:03:31.351 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.352 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.352 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:31.352 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.687866+0000", 2024-04-19T02:03:31.352 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0070097149999999997 2024-04-19T02:03:31.352 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.352 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.352 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:31.352 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.687913+0000", 2024-04-19T02:03:31.352 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.6878000000000001e-05 2024-04-19T02:03:31.352 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.352 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:31.352 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.352 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.353 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.353 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(osd.9.18:2 3.1s0 3:874b3480:::smithi05930959-36:head [assert-version v6,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:31.353 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.775686+0000", 2024-04-19T02:03:31.353 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 6.568972874, 2024-04-19T02:03:31.353 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010480552000000001, 2024-04-19T02:03:31.353 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:31.353 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:31.353 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:31.353 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:31.353 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.353 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:31.353 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.775686+0000", 2024-04-19T02:03:31.353 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.353 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.353 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.354 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:31.354 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.775686+0000", 2024-04-19T02:03:31.354 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.354 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.354 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.354 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:31.354 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.775687+0000", 2024-04-19T02:03:31.354 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3659999999999999e-06 2024-04-19T02:03:31.354 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.354 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.354 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:31.354 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.775694+0000", 2024-04-19T02:03:31.354 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.2540000000000003e-06 2024-04-19T02:03:31.355 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.355 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.355 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:31.355 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.775706+0000", 2024-04-19T02:03:31.355 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1654000000000001e-05 2024-04-19T02:03:31.355 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.355 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.355 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:31.355 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.775715+0000", 2024-04-19T02:03:31.355 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0189999999999995e-06 2024-04-19T02:03:31.355 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.355 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.355 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:31.355 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.775831+0000", 2024-04-19T02:03:31.355 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000115606 2024-04-19T02:03:31.356 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.356 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.356 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:31.356 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.776060+0000", 2024-04-19T02:03:31.356 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00022901700000000001 2024-04-19T02:03:31.356 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.356 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.356 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:31.356 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.786166+0000", 2024-04-19T02:03:31.356 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010106636 2024-04-19T02:03:31.356 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.356 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:31.356 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.356 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.357 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.357 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.ds7 46/26 ECSubWrite(tid=187, reqid=osd.8.0:187, at_version=46'35, trim_to=0'0, roll_forward_to=45'31))", 2024-04-19T02:03:31.357 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.058419+0000", 2024-04-19T02:03:31.357 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.286240037, 2024-04-19T02:03:31.357 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.007118699, 2024-04-19T02:03:31.357 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:31.357 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:31.357 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:31.357 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:31.357 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.357 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:31.357 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.058419+0000", 2024-04-19T02:03:31.357 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.357 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.358 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.358 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:31.358 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.058419+0000", 2024-04-19T02:03:31.358 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.358 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.358 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.358 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:31.358 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.058420+0000", 2024-04-19T02:03:31.358 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.9999999999999996e-07 2024-04-19T02:03:31.358 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.358 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.358 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:31.358 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.058454+0000", 2024-04-19T02:03:31.358 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4678000000000002e-05 2024-04-19T02:03:31.358 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.359 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.359 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:31.359 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.058474+0000", 2024-04-19T02:03:31.359 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9439000000000001e-05 2024-04-19T02:03:31.359 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.359 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.359 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:31.359 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.058485+0000", 2024-04-19T02:03:31.359 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1646e-05 2024-04-19T02:03:31.359 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.359 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.359 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:31.359 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.058684+0000", 2024-04-19T02:03:31.359 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00019885500000000001 2024-04-19T02:03:31.360 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.360 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.360 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:31.360 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.058697+0000", 2024-04-19T02:03:31.360 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2571999999999999e-05 2024-04-19T02:03:31.360 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.360 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.360 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:31.360 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.065499+0000", 2024-04-19T02:03:31.360 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0068021770000000004 2024-04-19T02:03:31.360 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.360 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.360 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:31.360 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.065537+0000", 2024-04-19T02:03:31.360 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.8532000000000003e-05 2024-04-19T02:03:31.361 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.361 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:31.361 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.361 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.361 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.361 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s7 46/26 ECSubWrite(tid=58, reqid=osd.1.0:58, at_version=46'44, trim_to=0'0, roll_forward_to=45'42))", 2024-04-19T02:03:31.361 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.059818+0000", 2024-04-19T02:03:31.361 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.284841039, 2024-04-19T02:03:31.361 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012903721, 2024-04-19T02:03:31.361 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:31.361 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:31.361 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:31.361 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:31.361 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.361 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:31.362 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.059818+0000", 2024-04-19T02:03:31.362 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.362 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.362 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.362 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:31.362 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.059818+0000", 2024-04-19T02:03:31.362 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.362 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.362 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.362 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:31.362 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.059818+0000", 2024-04-19T02:03:31.362 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4900000000000002e-07 2024-04-19T02:03:31.362 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.362 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.363 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:31.363 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.059841+0000", 2024-04-19T02:03:31.363 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3317999999999999e-05 2024-04-19T02:03:31.363 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.363 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.363 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:31.363 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.066536+0000", 2024-04-19T02:03:31.363 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0066946920000000004 2024-04-19T02:03:31.363 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.363 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.363 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:31.363 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.066552+0000", 2024-04-19T02:03:31.363 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5758999999999999e-05 2024-04-19T02:03:31.363 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.363 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.364 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:31.364 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.066653+0000", 2024-04-19T02:03:31.364 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000101648 2024-04-19T02:03:31.364 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.364 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.364 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:31.364 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.066667+0000", 2024-04-19T02:03:31.364 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3648999999999999e-05 2024-04-19T02:03:31.364 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.364 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.364 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:31.364 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.072672+0000", 2024-04-19T02:03:31.365 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0060046709999999996 2024-04-19T02:03:31.365 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.365 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.365 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:31.365 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.072721+0000", 2024-04-19T02:03:31.365 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9734999999999997e-05 2024-04-19T02:03:31.365 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.365 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:31.365 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.365 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.365 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.365 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:415 3.1s0 3:874b3480:::smithi05930959-36:head [append 0~720896] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:31.365 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.072647+0000", 2024-04-19T02:03:31.365 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.27201198, 2024-04-19T02:03:31.365 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.020543788, 2024-04-19T02:03:31.366 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:31.366 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:31.366 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:31.366 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:31.366 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:31.366 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:31.366 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 415 2024-04-19T02:03:31.366 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.366 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:31.367 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.367 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:31.367 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.072647+0000", 2024-04-19T02:03:31.367 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.367 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.367 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.367 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:31.367 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.072647+0000", 2024-04-19T02:03:31.367 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.367 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.367 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.367 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:31.367 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.072649+0000", 2024-04-19T02:03:31.367 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3649999999999998e-06 2024-04-19T02:03:31.368 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.368 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.368 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:31.368 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.073285+0000", 2024-04-19T02:03:31.368 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00063577000000000002 2024-04-19T02:03:31.368 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.368 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.368 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:31.368 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.073308+0000", 2024-04-19T02:03:31.368 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2612e-05 2024-04-19T02:03:31.368 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.368 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.368 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:31.368 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.073319+0000", 2024-04-19T02:03:31.368 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1992e-05 2024-04-19T02:03:31.368 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.369 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.369 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:31.369 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.073482+0000", 2024-04-19T02:03:31.369 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00016297800000000001 2024-04-19T02:03:31.369 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.369 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.369 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:31.369 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.073685+0000", 2024-04-19T02:03:31.369 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00020296299999999999 2024-04-19T02:03:31.369 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.369 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.369 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:31.369 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.076845+0000", 2024-04-19T02:03:31.369 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0031591380000000001 2024-04-19T02:03:31.369 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.369 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.370 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:31.370 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.085769+0000", 2024-04-19T02:03:31.370 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0089239320000000007 2024-04-19T02:03:31.370 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.370 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.370 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:31.370 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.093097+0000", 2024-04-19T02:03:31.370 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0073282130000000001 2024-04-19T02:03:31.370 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.370 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.370 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:31.370 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.093191+0000", 2024-04-19T02:03:31.370 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3825000000000004e-05 2024-04-19T02:03:31.370 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.371 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:31.371 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.371 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.371 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.371 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.7s3 46/26 ECSubWrite(tid=158, reqid=osd.3.0:158, at_version=46'14, trim_to=0'0, roll_forward_to=45'12))", 2024-04-19T02:03:31.371 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.073239+0000", 2024-04-19T02:03:31.371 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.2714193890000001, 2024-04-19T02:03:31.371 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0060590260000000003, 2024-04-19T02:03:31.371 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:31.371 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:31.371 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:31.371 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:31.371 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.371 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:31.371 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.073239+0000", 2024-04-19T02:03:31.372 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.372 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.372 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.372 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:31.372 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.073239+0000", 2024-04-19T02:03:31.372 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.372 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.372 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.372 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:31.372 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.073240+0000", 2024-04-19T02:03:31.372 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3300000000000001e-07 2024-04-19T02:03:31.372 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.372 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.372 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:31.372 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.073273+0000", 2024-04-19T02:03:31.373 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3889e-05 2024-04-19T02:03:31.373 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.373 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.373 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:31.373 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.073287+0000", 2024-04-19T02:03:31.373 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3559e-05 2024-04-19T02:03:31.373 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.373 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.373 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:31.373 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.073299+0000", 2024-04-19T02:03:31.373 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1505000000000001e-05 2024-04-19T02:03:31.373 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.373 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.373 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:31.374 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.073370+0000", 2024-04-19T02:03:31.374 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.1501000000000005e-05 2024-04-19T02:03:31.374 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.374 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.374 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:31.374 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.073382+0000", 2024-04-19T02:03:31.374 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1759999999999999e-05 2024-04-19T02:03:31.374 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.374 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.374 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:31.374 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.079253+0000", 2024-04-19T02:03:31.374 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0058712349999999998 2024-04-19T02:03:31.374 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.374 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.375 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:31.375 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.079298+0000", 2024-04-19T02:03:31.375 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5343999999999998e-05 2024-04-19T02:03:31.375 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.375 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:31.375 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.375 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.375 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.375 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:416 3.1s0 3:874b3480:::smithi05930959-36:head [append 0~442368] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:31.375 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.077896+0000", 2024-04-19T02:03:31.375 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.2667628209999999, 2024-04-19T02:03:31.375 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.018242536, 2024-04-19T02:03:31.375 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:31.375 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:31.375 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:31.376 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:31.376 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:31.376 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:31.376 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 416 2024-04-19T02:03:31.376 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.376 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:31.376 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.376 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:31.376 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077896+0000", 2024-04-19T02:03:31.376 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.376 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.376 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.376 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:31.376 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077896+0000", 2024-04-19T02:03:31.377 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.377 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.377 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.377 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:31.377 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077898+0000", 2024-04-19T02:03:31.377 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1959999999999998e-06 2024-04-19T02:03:31.377 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.377 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.377 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:31.377 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.078421+0000", 2024-04-19T02:03:31.377 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00052314500000000003 2024-04-19T02:03:31.377 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.377 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.378 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:31.378 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.078441+0000", 2024-04-19T02:03:31.378 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9763999999999999e-05 2024-04-19T02:03:31.378 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.378 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.378 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:31.378 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.078453+0000", 2024-04-19T02:03:31.378 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1724e-05 2024-04-19T02:03:31.378 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.378 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.378 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:31.378 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.079580+0000", 2024-04-19T02:03:31.378 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0011274550000000001 2024-04-19T02:03:31.378 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.379 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.379 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:31.379 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.079689+0000", 2024-04-19T02:03:31.379 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108942 2024-04-19T02:03:31.379 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.379 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.379 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:31.379 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080934+0000", 2024-04-19T02:03:31.379 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001245187 2024-04-19T02:03:31.379 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.379 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.379 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:31.379 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.085771+0000", 2024-04-19T02:03:31.379 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.004836707 2024-04-19T02:03:31.379 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.380 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.380 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:31.380 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.096060+0000", 2024-04-19T02:03:31.380 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010288976 2024-04-19T02:03:31.380 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.380 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.380 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:31.380 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.096138+0000", 2024-04-19T02:03:31.380 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.8440000000000001e-05 2024-04-19T02:03:31.380 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.380 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:31.380 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.380 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.381 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.381 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:417 3.1s0 3:874b3480:::smithi05930959-36:head [append 0~327680] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:31.381 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.079192+0000", 2024-04-19T02:03:31.381 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.2654662910000001, 2024-04-19T02:03:31.381 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.01712928, 2024-04-19T02:03:31.381 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:31.381 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:31.381 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:31.381 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:31.381 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:31.381 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:31.381 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 417 2024-04-19T02:03:31.381 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.381 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:31.381 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.382 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:31.382 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.079192+0000", 2024-04-19T02:03:31.382 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.382 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.382 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.382 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:31.382 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.079192+0000", 2024-04-19T02:03:31.382 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.382 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.382 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.382 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:31.382 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.079195+0000", 2024-04-19T02:03:31.382 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1880000000000001e-06 2024-04-19T02:03:31.382 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.383 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.383 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:31.383 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.079506+0000", 2024-04-19T02:03:31.383 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00031092000000000001 2024-04-19T02:03:31.383 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.383 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.383 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:31.383 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.079526+0000", 2024-04-19T02:03:31.383 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0404e-05 2024-04-19T02:03:31.383 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.383 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.383 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:31.383 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.079540+0000", 2024-04-19T02:03:31.383 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4233000000000001e-05 2024-04-19T02:03:31.384 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.384 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.384 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:31.384 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.081867+0000", 2024-04-19T02:03:31.384 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0023265030000000002 2024-04-19T02:03:31.384 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.384 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.384 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:31.384 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.082009+0000", 2024-04-19T02:03:31.384 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014187299999999999 2024-04-19T02:03:31.384 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.384 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.384 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:31.384 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.083078+0000", 2024-04-19T02:03:31.384 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0010689499999999999 2024-04-19T02:03:31.385 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.385 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.385 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:31.385 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.085773+0000", 2024-04-19T02:03:31.385 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002695771 2024-04-19T02:03:31.385 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.385 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.385 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:31.385 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.096256+0000", 2024-04-19T02:03:31.385 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010482551 2024-04-19T02:03:31.385 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.385 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.385 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:31.385 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.096322+0000", 2024-04-19T02:03:31.386 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.5887000000000001e-05 2024-04-19T02:03:31.386 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.386 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:31.386 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.386 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.386 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.386 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:418 3.1s0 3:874b3480:::smithi05930959-36:head [setxattr _header (56) in=7b] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:31.386 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.080040+0000", 2024-04-19T02:03:31.386 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.2646182910000001, 2024-04-19T02:03:31.386 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.016869248, 2024-04-19T02:03:31.386 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:31.386 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:31.386 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:31.386 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:31.387 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:31.387 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:31.387 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 418 2024-04-19T02:03:31.387 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.387 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:31.387 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.387 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:31.387 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080040+0000", 2024-04-19T02:03:31.387 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.387 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.387 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.387 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:31.387 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080040+0000", 2024-04-19T02:03:31.387 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.388 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.388 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.388 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:31.388 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080042+0000", 2024-04-19T02:03:31.388 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.215e-06 2024-04-19T02:03:31.388 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.388 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.388 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:31.388 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080049+0000", 2024-04-19T02:03:31.388 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.3180000000000001e-06 2024-04-19T02:03:31.388 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.388 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.388 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:31.388 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080067+0000", 2024-04-19T02:03:31.388 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7805999999999999e-05 2024-04-19T02:03:31.388 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.389 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.389 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:31.389 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080078+0000", 2024-04-19T02:03:31.389 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1262e-05 2024-04-19T02:03:31.389 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.389 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.389 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:31.389 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.084057+0000", 2024-04-19T02:03:31.389 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0039794039999999998 2024-04-19T02:03:31.389 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.389 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.389 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:31.389 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.084161+0000", 2024-04-19T02:03:31.389 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000103862 2024-04-19T02:03:31.389 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.389 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.390 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:31.390 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.084626+0000", 2024-04-19T02:03:31.390 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00046457200000000001 2024-04-19T02:03:31.390 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.390 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.390 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:31.390 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.086690+0000", 2024-04-19T02:03:31.390 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0020638119999999999 2024-04-19T02:03:31.390 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.390 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.390 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:31.390 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.096836+0000", 2024-04-19T02:03:31.390 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.01014584 2024-04-19T02:03:31.391 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.391 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.391 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:31.391 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.096910+0000", 2024-04-19T02:03:31.391 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.4157000000000002e-05 2024-04-19T02:03:31.391 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.391 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:31.391 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.391 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.391 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.391 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:419 3.1s0 3:874b3480:::smithi05930959-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:31.391 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.080096+0000", 2024-04-19T02:03:31.391 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.2645630670000001, 2024-04-19T02:03:31.391 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.019302687999999998, 2024-04-19T02:03:31.391 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:31.391 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:31.392 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:31.392 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:31.392 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:31.392 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:31.392 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 419 2024-04-19T02:03:31.392 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.392 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:31.392 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.392 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:31.392 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080096+0000", 2024-04-19T02:03:31.392 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.392 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.392 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.392 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:31.393 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080096+0000", 2024-04-19T02:03:31.393 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.393 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.393 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.393 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:31.393 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080096+0000", 2024-04-19T02:03:31.393 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.1200000000000002e-07 2024-04-19T02:03:31.393 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.393 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.393 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:31.393 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080100+0000", 2024-04-19T02:03:31.393 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.506e-06 2024-04-19T02:03:31.393 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.393 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.394 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:31.394 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080111+0000", 2024-04-19T02:03:31.394 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0879e-05 2024-04-19T02:03:31.394 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.394 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.394 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:31.394 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080119+0000", 2024-04-19T02:03:31.394 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.9629999999999998e-06 2024-04-19T02:03:31.394 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.394 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.394 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:31.394 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.085126+0000", 2024-04-19T02:03:31.394 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0050074280000000004 2024-04-19T02:03:31.394 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.394 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.395 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:31.395 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.085230+0000", 2024-04-19T02:03:31.395 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000103589 2024-04-19T02:03:31.395 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.395 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.395 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:31.395 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.097332+0000", 2024-04-19T02:03:31.395 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012101796999999999 2024-04-19T02:03:31.395 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.395 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.395 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:31.395 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.097477+0000", 2024-04-19T02:03:31.395 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014539899999999999 2024-04-19T02:03:31.395 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.395 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.396 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:31.396 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.099398+0000", 2024-04-19T02:03:31.396 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0019214150000000001 2024-04-19T02:03:31.396 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.396 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:31.396 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.396 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.396 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.396 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(osd.3.18:3 3.1s0 3:874b3480:::smithi05930959-36:head [assert-version v11,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:31.396 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.195845+0000", 2024-04-19T02:03:31.396 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.148813428, 2024-04-19T02:03:31.396 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012426504, 2024-04-19T02:03:31.396 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:31.396 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:31.396 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:31.397 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:31.397 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.397 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:31.397 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195845+0000", 2024-04-19T02:03:31.397 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.397 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.397 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.397 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:31.397 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195845+0000", 2024-04-19T02:03:31.397 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.397 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.397 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.397 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:31.397 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195847+0000", 2024-04-19T02:03:31.397 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0090000000000002e-06 2024-04-19T02:03:31.398 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.398 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.398 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:31.398 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195856+0000", 2024-04-19T02:03:31.398 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5650000000000002e-06 2024-04-19T02:03:31.398 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.398 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.398 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:31.398 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195871+0000", 2024-04-19T02:03:31.398 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5072000000000001e-05 2024-04-19T02:03:31.398 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.398 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.398 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:31.398 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195879+0000", 2024-04-19T02:03:31.399 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3680000000000002e-06 2024-04-19T02:03:31.399 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.399 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.399 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:31.399 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195978+0000", 2024-04-19T02:03:31.399 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.8202000000000006e-05 2024-04-19T02:03:31.399 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.399 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.399 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:31.399 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.196085+0000", 2024-04-19T02:03:31.399 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000107506 2024-04-19T02:03:31.399 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.399 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.399 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:31.399 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.208272+0000", 2024-04-19T02:03:31.400 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012186782 2024-04-19T02:03:31.400 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.400 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:31.400 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.400 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.400 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.400 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:447 3.1s0 3:88463a85:::smithi05930959-33:head [stat,delete] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:31.400 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.226277+0000", 2024-04-19T02:03:31.400 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 1.1183820600000001, 2024-04-19T02:03:31.400 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.008527494, 2024-04-19T02:03:31.400 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:31.400 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:31.400 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:31.400 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:31.401 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:31.401 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:31.401 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 447 2024-04-19T02:03:31.401 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.401 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:31.401 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.401 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:31.401 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.226277+0000", 2024-04-19T02:03:31.401 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.401 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.402 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.402 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:31.402 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.226277+0000", 2024-04-19T02:03:31.402 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.402 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.402 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.402 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:31.402 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.226279+0000", 2024-04-19T02:03:31.402 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.526e-06 2024-04-19T02:03:31.402 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.402 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.402 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:31.402 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.226293+0000", 2024-04-19T02:03:31.402 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4093e-05 2024-04-19T02:03:31.402 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.403 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.403 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:31.403 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.226316+0000", 2024-04-19T02:03:31.403 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2875e-05 2024-04-19T02:03:31.403 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.403 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.403 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:31.403 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.226331+0000", 2024-04-19T02:03:31.403 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4677e-05 2024-04-19T02:03:31.403 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.403 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.403 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:31.403 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.226421+0000", 2024-04-19T02:03:31.403 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0027999999999999e-05 2024-04-19T02:03:31.403 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.404 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.404 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:31.404 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.226534+0000", 2024-04-19T02:03:31.404 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000113504 2024-04-19T02:03:31.404 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.406 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.406 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:31.406 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.227340+0000", 2024-04-19T02:03:31.406 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00080527699999999997 2024-04-19T02:03:31.406 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.406 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.406 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:31.406 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.228989+0000", 2024-04-19T02:03:31.406 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0016491940000000001 2024-04-19T02:03:31.406 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.406 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.406 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:31.406 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.234694+0000", 2024-04-19T02:03:31.407 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0057055960000000003 2024-04-19T02:03:31.407 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.407 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.407 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:31.407 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.234804+0000", 2024-04-19T02:03:31.407 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000109724 2024-04-19T02:03:31.407 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.407 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:31.407 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.407 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.407 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.407 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs1 47/26 ECSubWrite(tid=180, reqid=osd.3.0:180, at_version=47'36, trim_to=0'0, roll_forward_to=46'34))", 2024-04-19T02:03:31.407 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:31.080178+0000", 2024-04-19T02:03:31.407 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.26448064900000001, 2024-04-19T02:03:31.408 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0072941200000000003, 2024-04-19T02:03:31.408 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:31.408 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:31.408 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:31.408 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:31.408 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.408 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:31.408 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.080178+0000", 2024-04-19T02:03:31.408 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.408 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.408 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.408 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:31.408 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.080178+0000", 2024-04-19T02:03:31.408 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.408 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.408 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.409 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:31.409 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.080179+0000", 2024-04-19T02:03:31.409 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.44e-07 2024-04-19T02:03:31.409 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.409 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.409 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:31.409 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.080216+0000", 2024-04-19T02:03:31.409 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.7452e-05 2024-04-19T02:03:31.409 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.409 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.409 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:31.409 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.080245+0000", 2024-04-19T02:03:31.409 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9252999999999999e-05 2024-04-19T02:03:31.409 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.409 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.410 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:31.410 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.080277+0000", 2024-04-19T02:03:31.410 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2008999999999998e-05 2024-04-19T02:03:31.410 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.410 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.410 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:31.410 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.080366+0000", 2024-04-19T02:03:31.410 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.9118999999999998e-05 2024-04-19T02:03:31.410 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.410 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.410 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:31.410 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.080380+0000", 2024-04-19T02:03:31.410 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3580999999999999e-05 2024-04-19T02:03:31.410 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.410 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.411 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:31.411 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.087436+0000", 2024-04-19T02:03:31.411 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0070562699999999999 2024-04-19T02:03:31.411 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.411 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.411 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:31.411 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.087472+0000", 2024-04-19T02:03:31.411 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5892000000000001e-05 2024-04-19T02:03:31.411 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.411 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:31.411 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.411 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.411 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.411 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs1 47/26 ECSubWrite(tid=183, reqid=osd.3.0:183, at_version=47'38, trim_to=0'0, roll_forward_to=46'34))", 2024-04-19T02:03:31.411 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:31.081854+0000", 2024-04-19T02:03:31.411 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.26280455800000002, 2024-04-19T02:03:31.412 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0058817640000000003, 2024-04-19T02:03:31.412 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:31.412 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:31.412 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:31.412 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:31.412 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.412 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:31.412 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.081854+0000", 2024-04-19T02:03:31.412 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.412 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.412 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.412 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:31.412 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.081854+0000", 2024-04-19T02:03:31.412 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.412 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.413 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.413 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:31.413 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.081855+0000", 2024-04-19T02:03:31.413 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.206e-06 2024-04-19T02:03:31.413 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.413 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.413 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:31.413 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.081885+0000", 2024-04-19T02:03:31.413 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9369999999999998e-05 2024-04-19T02:03:31.413 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.413 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.413 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:31.413 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.081902+0000", 2024-04-19T02:03:31.413 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6792000000000001e-05 2024-04-19T02:03:31.413 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.414 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.414 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:31.414 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.081934+0000", 2024-04-19T02:03:31.414 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2656e-05 2024-04-19T02:03:31.414 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.414 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.414 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:31.414 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.085476+0000", 2024-04-19T02:03:31.414 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0035419050000000001 2024-04-19T02:03:31.414 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.414 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.414 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:31.414 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.085493+0000", 2024-04-19T02:03:31.415 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7158000000000001e-05 2024-04-19T02:03:31.415 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.415 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.415 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:31.415 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.087707+0000", 2024-04-19T02:03:31.415 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002213919 2024-04-19T02:03:31.415 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.415 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.415 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:31.415 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.087736+0000", 2024-04-19T02:03:31.415 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8758000000000001e-05 2024-04-19T02:03:31.415 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.415 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:31.415 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.415 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.415 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.416 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:465 3.1s0 3:874b3480:::smithi05930959-36:head [append 0~786432] snapc a=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:03:31.416 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:31.156851+0000", 2024-04-19T02:03:31.416 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.18780748999999999, 2024-04-19T02:03:31.416 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0094295679999999993, 2024-04-19T02:03:31.416 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:31.416 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:31.416 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:31.416 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:31.416 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:31.416 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:31.416 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 465 2024-04-19T02:03:31.416 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.416 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:31.416 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.416 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:31.417 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.156851+0000", 2024-04-19T02:03:31.417 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.417 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.417 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.417 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:31.417 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.156851+0000", 2024-04-19T02:03:31.417 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.417 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.417 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.417 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:31.417 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.156854+0000", 2024-04-19T02:03:31.417 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8030000000000001e-06 2024-04-19T02:03:31.417 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.417 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.418 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:31.418 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.157331+0000", 2024-04-19T02:03:31.418 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00047709100000000002 2024-04-19T02:03:31.418 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.418 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.418 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:31.418 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.157357+0000", 2024-04-19T02:03:31.418 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6092e-05 2024-04-19T02:03:31.418 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.418 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.418 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:31.418 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.157371+0000", 2024-04-19T02:03:31.418 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.403e-05 2024-04-19T02:03:31.418 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.418 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.419 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:31.419 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.157471+0000", 2024-04-19T02:03:31.419 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000100049 2024-04-19T02:03:31.419 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.419 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.419 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:31.419 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.157607+0000", 2024-04-19T02:03:31.419 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013581500000000001 2024-04-19T02:03:31.419 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.419 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.419 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:31.419 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.159060+0000", 2024-04-19T02:03:31.419 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001452971 2024-04-19T02:03:31.419 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.419 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.420 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:31.420 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.160520+0000", 2024-04-19T02:03:31.420 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0014597449999999999 2024-04-19T02:03:31.420 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.420 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.420 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:31.420 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.166184+0000", 2024-04-19T02:03:31.420 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0056638799999999996 2024-04-19T02:03:31.420 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.420 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.420 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:31.420 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.166281+0000", 2024-04-19T02:03:31.420 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.7091999999999998e-05 2024-04-19T02:03:31.420 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.420 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:31.421 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.421 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.421 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.421 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:466 3.1s0 3:874b3480:::smithi05930959-36:head [append 0~491520] snapc a=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:03:31.421 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:31.160285+0000", 2024-04-19T02:03:31.421 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.1843737, 2024-04-19T02:03:31.421 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0066126589999999999, 2024-04-19T02:03:31.421 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:31.421 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:31.421 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:31.421 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:31.421 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:31.421 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:31.421 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 466 2024-04-19T02:03:31.421 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.422 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:31.422 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.422 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:31.422 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.160285+0000", 2024-04-19T02:03:31.422 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.422 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.422 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.422 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:31.422 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.160285+0000", 2024-04-19T02:03:31.422 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.422 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.422 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.422 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:31.422 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.160287+0000", 2024-04-19T02:03:31.422 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.857e-06 2024-04-19T02:03:31.423 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.423 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.423 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:31.423 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.160603+0000", 2024-04-19T02:03:31.423 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000316008 2024-04-19T02:03:31.423 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.423 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.423 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:31.423 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.160620+0000", 2024-04-19T02:03:31.423 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6977e-05 2024-04-19T02:03:31.423 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.423 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.423 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:31.423 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.160631+0000", 2024-04-19T02:03:31.423 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1403000000000001e-05 2024-04-19T02:03:31.424 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.424 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.424 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:31.424 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.160715+0000", 2024-04-19T02:03:31.424 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3820999999999996e-05 2024-04-19T02:03:31.424 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.424 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.424 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:31.424 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.160824+0000", 2024-04-19T02:03:31.424 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108744 2024-04-19T02:03:31.424 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.424 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.424 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:31.424 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.161896+0000", 2024-04-19T02:03:31.424 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001071916 2024-04-19T02:03:31.424 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.425 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.425 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:31.425 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.165411+0000", 2024-04-19T02:03:31.425 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0035147720000000002 2024-04-19T02:03:31.425 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.425 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.425 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:31.425 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.166822+0000", 2024-04-19T02:03:31.425 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0014117820000000001 2024-04-19T02:03:31.425 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.425 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.425 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:31.425 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.166898+0000", 2024-04-19T02:03:31.425 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.5378999999999998e-05 2024-04-19T02:03:31.425 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.426 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:31.426 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.426 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.426 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.426 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:467 3.1s0 3:874b3480:::smithi05930959-36:head [append 0~278528] snapc a=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:03:31.426 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:31.162400+0000", 2024-04-19T02:03:31.426 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.18225865099999999, 2024-04-19T02:03:31.426 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0057940999999999999, 2024-04-19T02:03:31.426 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:31.426 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:31.426 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:31.426 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:31.426 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:31.426 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:31.426 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 467 2024-04-19T02:03:31.427 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.427 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:31.427 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.427 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:31.427 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162400+0000", 2024-04-19T02:03:31.427 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.427 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.427 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.427 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:31.427 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162400+0000", 2024-04-19T02:03:31.427 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.427 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.427 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.427 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:31.427 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162402+0000", 2024-04-19T02:03:31.428 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.598e-06 2024-04-19T02:03:31.428 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.428 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.428 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:31.428 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162524+0000", 2024-04-19T02:03:31.428 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000121828 2024-04-19T02:03:31.428 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.428 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.428 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:31.428 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162537+0000", 2024-04-19T02:03:31.428 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.385e-05 2024-04-19T02:03:31.428 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.428 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.428 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:31.429 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162545+0000", 2024-04-19T02:03:31.429 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.7889999999999995e-06 2024-04-19T02:03:31.429 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.429 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.429 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:31.429 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162974+0000", 2024-04-19T02:03:31.429 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000428503 2024-04-19T02:03:31.429 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.429 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.429 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:31.429 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.163104+0000", 2024-04-19T02:03:31.429 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013012200000000001 2024-04-19T02:03:31.429 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.429 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.430 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:31.430 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.164086+0000", 2024-04-19T02:03:31.430 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00098271600000000001 2024-04-19T02:03:31.430 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.430 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.430 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:31.430 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.165462+0000", 2024-04-19T02:03:31.430 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001375802 2024-04-19T02:03:31.430 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.430 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.430 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:31.430 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.168112+0000", 2024-04-19T02:03:31.430 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0026497399999999998 2024-04-19T02:03:31.430 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.431 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.431 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:31.431 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.168194+0000", 2024-04-19T02:03:31.431 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2151999999999995e-05 2024-04-19T02:03:31.431 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.431 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:31.431 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.431 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.431 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.431 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:468 3.1s0 3:874b3480:::smithi05930959-36:head [setxattr _header (56) in=7b] snapc a=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:03:31.431 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:31.162561+0000", 2024-04-19T02:03:31.431 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.18209771299999999, 2024-04-19T02:03:31.431 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0065321470000000003, 2024-04-19T02:03:31.431 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:31.431 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:31.432 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:31.432 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:31.432 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:31.432 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:31.432 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 468 2024-04-19T02:03:31.432 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.432 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:31.432 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.432 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:31.432 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162561+0000", 2024-04-19T02:03:31.432 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.432 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.432 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.432 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:31.433 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162561+0000", 2024-04-19T02:03:31.433 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.433 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.433 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.433 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:31.433 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162562+0000", 2024-04-19T02:03:31.433 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.5199999999999996e-07 2024-04-19T02:03:31.433 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.433 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.433 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:31.433 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162566+0000", 2024-04-19T02:03:31.433 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0160000000000002e-06 2024-04-19T02:03:31.433 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.433 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.433 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:31.434 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162574+0000", 2024-04-19T02:03:31.434 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5569999999999996e-06 2024-04-19T02:03:31.434 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.434 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.434 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:31.434 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162580+0000", 2024-04-19T02:03:31.434 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.1890000000000004e-06 2024-04-19T02:03:31.434 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.434 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.434 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:31.434 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.164615+0000", 2024-04-19T02:03:31.434 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002034414 2024-04-19T02:03:31.434 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.434 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.435 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:31.435 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.164704+0000", 2024-04-19T02:03:31.435 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.9485000000000001e-05 2024-04-19T02:03:31.435 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.435 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.435 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:31.435 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.165220+0000", 2024-04-19T02:03:31.435 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00051539499999999998 2024-04-19T02:03:31.435 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.435 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.435 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:31.435 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.165882+0000", 2024-04-19T02:03:31.435 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00066206499999999998 2024-04-19T02:03:31.435 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.435 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.436 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:31.436 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.169006+0000", 2024-04-19T02:03:31.436 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0031242050000000001 2024-04-19T02:03:31.436 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.436 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.436 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:31.436 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.169093+0000", 2024-04-19T02:03:31.436 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7169000000000005e-05 2024-04-19T02:03:31.436 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.436 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:31.436 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.436 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.436 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.436 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:469 3.1s0 3:874b3480:::smithi05930959-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:03:31.436 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:31.162590+0000", 2024-04-19T02:03:31.437 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.18206863500000001, 2024-04-19T02:03:31.437 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0086854280000000002, 2024-04-19T02:03:31.437 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:31.437 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:31.437 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:31.437 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:31.437 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:31.437 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:31.437 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 469 2024-04-19T02:03:31.437 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.437 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:31.437 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.437 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:31.437 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162590+0000", 2024-04-19T02:03:31.437 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.438 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.438 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.438 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:31.438 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162590+0000", 2024-04-19T02:03:31.438 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:31.438 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.438 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.438 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:31.438 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162591+0000", 2024-04-19T02:03:31.438 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.1600000000000001e-07 2024-04-19T02:03:31.438 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.438 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.438 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:31.438 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162593+0000", 2024-04-19T02:03:31.438 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.458e-06 2024-04-19T02:03:31.439 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.439 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.439 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:31.439 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162604+0000", 2024-04-19T02:03:31.439 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0869e-05 2024-04-19T02:03:31.439 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.439 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.439 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:31.439 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162610+0000", 2024-04-19T02:03:31.439 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.2020000000000001e-06 2024-04-19T02:03:31.439 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.439 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.439 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:31.439 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.165692+0000", 2024-04-19T02:03:31.440 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0030820650000000002 2024-04-19T02:03:31.440 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.440 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.440 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:31.440 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.165778+0000", 2024-04-19T02:03:31.440 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5927000000000006e-05 2024-04-19T02:03:31.440 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.440 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.440 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:31.440 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.169491+0000", 2024-04-19T02:03:31.440 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0037127610000000002 2024-04-19T02:03:31.440 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.440 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.440 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:31.441 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.169588+0000", 2024-04-19T02:03:31.441 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.6850999999999998e-05 2024-04-19T02:03:31.441 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:31.441 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:31.441 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:31.441 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.171276+0000", 2024-04-19T02:03:31.441 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0016877789999999999 2024-04-19T02:03:31.441 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.441 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:31.441 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.441 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:31.441 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:31.441 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:31.443 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-19T02:03:31.444 INFO:tasks.rados.rados.0.smithi059.stdout:186: finishing write tid 1 to smithi05930959-19 2024-04-19T02:03:31.444 INFO:tasks.rados.rados.0.smithi059.stdout:186: finishing write tid 2 to smithi05930959-19 2024-04-19T02:03:31.444 INFO:tasks.rados.rados.0.smithi059.stdout:186: finishing write tid 3 to smithi05930959-19 2024-04-19T02:03:31.444 INFO:tasks.rados.rados.0.smithi059.stdout:186: finishing write tid 4 to smithi05930959-19 2024-04-19T02:03:31.444 INFO:tasks.rados.rados.0.smithi059.stdout:186: finishing write tid 5 to smithi05930959-19 2024-04-19T02:03:31.444 INFO:tasks.rados.rados.0.smithi059.stdout:186: oid 19 updating version 0 to 45 2024-04-19T02:03:31.444 INFO:tasks.rados.rados.0.smithi059.stdout:186: oid 19 version 45 is already newer than 42 2024-04-19T02:03:31.444 INFO:tasks.rados.rados.0.smithi059.stdout:186: oid 19 version 45 is already newer than 43 2024-04-19T02:03:31.444 INFO:tasks.rados.rados.0.smithi059.stdout:186: oid 19 version 45 is already newer than 44 2024-04-19T02:03:31.444 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 19 v 45 (ObjNum 96 snap 9 seq_num 96) dirty exists 2024-04-19T02:03:31.444 INFO:tasks.rados.rados.0.smithi059.stdout:186: left oid 19 (ObjNum 96 snap 9 seq_num 96) 2024-04-19T02:03:31.445 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:31.442+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:31.543 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:31.542+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:31.615 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:31.615 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:03:31.615 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-04-19T02:03:31.615 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:31.629 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-19T02:03:31.645 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:31.646+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:31.681 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T02:03:31.758 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:31.758+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:31.797 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:31.797 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:03:31.797 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-04-19T02:03:31.797 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-04-19T02:03:31.797 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:31.811 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-19T02:03:31.847 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:03:31.846+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:03:31.860 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-04-19T02:03:31.860 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T02:03:31.875 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-04-19T02:03:31.876 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T02:03:31.892 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-04-19T02:03:31.892 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T02:03:31.908 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-04-19T02:03:31.909 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T02:03:31.929 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-04-19T02:03:31.929 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T02:03:31.950 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:31.946+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:31.951 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-04-19T02:03:31.951 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T02:03:31.967 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:31.967 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:31.972 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-04-19T02:03:31.972 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T02:03:31.980 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-19T02:03:31.993 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-04-19T02:03:31.994 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T02:03:32.015 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-04-19T02:03:32.015 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T02:03:32.036 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.5 to 1 (10000) 2024-04-19T02:03:32.036 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-04-19T02:03:32.036 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T02:03:32.043 INFO:tasks.rados.rados.0.smithi059.stdout:176: done (10 left) 2024-04-19T02:03:32.043 INFO:tasks.rados.rados.0.smithi059.stdout:177: done (9 left) 2024-04-19T02:03:32.043 INFO:tasks.rados.rados.0.smithi059.stdout:178: done (8 left) 2024-04-19T02:03:32.043 INFO:tasks.rados.rados.0.smithi059.stdout:179: done (7 left) 2024-04-19T02:03:32.043 INFO:tasks.rados.rados.0.smithi059.stdout:180: done (6 left) 2024-04-19T02:03:32.043 INFO:tasks.rados.rados.0.smithi059.stdout:181: done (5 left) 2024-04-19T02:03:32.044 INFO:tasks.rados.rados.0.smithi059.stdout:182: done (4 left) 2024-04-19T02:03:32.044 INFO:tasks.rados.rados.0.smithi059.stdout:183: done (3 left) 2024-04-19T02:03:32.044 INFO:tasks.rados.rados.0.smithi059.stdout:185: done (2 left) 2024-04-19T02:03:32.044 INFO:tasks.rados.rados.0.smithi059.stdout:186: done (1 left) 2024-04-19T02:03:32.044 INFO:tasks.rados.rados.0.smithi059.stdout:189: done (0 left) 2024-04-19T02:03:32.044 INFO:tasks.rados.rados.0.smithi059.stdout:190: read oid 11 snap -1 2024-04-19T02:03:32.044 INFO:tasks.rados.rados.0.smithi059.stdout:190: expect (ObjNum 10 snap 0 seq_num 10) 2024-04-19T02:03:32.044 INFO:tasks.rados.rados.0.smithi059.stdout:191: rmattr oid 1 current snap is 10 2024-04-19T02:03:32.044 INFO:tasks.rados.rados.0.smithi059.stdout:191: done (1 left) 2024-04-19T02:03:32.044 INFO:tasks.rados.rados.0.smithi059.stdout:192: read oid 7 snap 9 2024-04-19T02:03:32.044 INFO:tasks.rados.rados.0.smithi059.stdout:192: expect (ObjNum 91 snap 9 seq_num 91) 2024-04-19T02:03:32.044 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 46 current snap is 10 2024-04-19T02:03:32.044 INFO:tasks.rados.rados.0.smithi059.stdout:193: seq_num 97 ranges {770048=475136,1245184=442368,1687552=442368,2129920=278528} 2024-04-19T02:03:32.051 INFO:tasks.rados.rados.0.smithi059.stdout:193: writing smithi05930959-46 from 770048 to 1245184 tid 1 2024-04-19T02:03:32.052 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:32.050+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:32.055 INFO:tasks.rados.rados.0.smithi059.stdout:193: writing smithi05930959-46 from 1245184 to 1687552 tid 2 2024-04-19T02:03:32.059 INFO:tasks.rados.rados.0.smithi059.stdout:193: writing smithi05930959-46 from 1687552 to 2129920 tid 3 2024-04-19T02:03:32.060 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-04-19T02:03:32.061 INFO:tasks.rados.rados.0.smithi059.stdout:193: writing smithi05930959-46 from 2129920 to 2408448 tid 4 2024-04-19T02:03:32.061 INFO:tasks.rados.rados.0.smithi059.stdout:194: copy_from oid 17 from oid 15 current snap is 10 2024-04-19T02:03:32.061 INFO:tasks.rados.rados.0.smithi059.stdout:190: expect (ObjNum 10 snap 0 seq_num 10) 2024-04-19T02:03:32.068 INFO:tasks.rados.rados.0.smithi059.stdout:190: done (3 left) 2024-04-19T02:03:32.068 INFO:tasks.rados.rados.0.smithi059.stdout:195: snap_create 2024-04-19T02:03:32.072 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-04-19T02:03:32.072 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T02:03:32.080 INFO:tasks.rados.rados.0.smithi059.stdout:194: finishing copy_from to smithi05930959-17 2024-04-19T02:03:32.080 INFO:tasks.rados.rados.0.smithi059.stdout:194: got expected ENOENT (src dne) 2024-04-19T02:03:32.080 INFO:tasks.rados.rados.0.smithi059.stdout:194: finishing copy_from racing read to smithi05930959-17 2024-04-19T02:03:32.083 INFO:tasks.rados.rados.0.smithi059.stdout:192: expect (ObjNum 91 snap 9 seq_num 91) 2024-04-19T02:03:32.099 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-19T02:03:32.112 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-04-19T02:03:32.113 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T02:03:32.131 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:32.131 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:32.152 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-19T02:03:32.153 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:32.150+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:32.196 INFO:tasks.rados.rados.0.smithi059.stdout:193: finishing write tid 1 to smithi05930959-46 2024-04-19T02:03:32.197 INFO:tasks.rados.rados.0.smithi059.stdout:193: finishing write tid 2 to smithi05930959-46 2024-04-19T02:03:32.197 INFO:tasks.rados.rados.0.smithi059.stdout:193: finishing write tid 3 to smithi05930959-46 2024-04-19T02:03:32.197 INFO:tasks.rados.rados.0.smithi059.stdout:193: finishing write tid 4 to smithi05930959-46 2024-04-19T02:03:32.197 INFO:tasks.rados.rados.0.smithi059.stdout:193: finishing write tid 5 to smithi05930959-46 2024-04-19T02:03:32.197 INFO:tasks.rados.rados.0.smithi059.stdout:193: finishing write tid 6 to smithi05930959-46 2024-04-19T02:03:32.197 INFO:tasks.rados.rados.0.smithi059.stdout:193: oid 46 updating version 0 to 13 2024-04-19T02:03:32.197 INFO:tasks.rados.rados.0.smithi059.stdout:193: oid 46 version 13 is already newer than 12 2024-04-19T02:03:32.197 INFO:tasks.rados.rados.0.smithi059.stdout:193: oid 46 updating version 13 to 14 2024-04-19T02:03:32.197 INFO:tasks.rados.rados.0.smithi059.stdout:193: oid 46 updating version 14 to 15 2024-04-19T02:03:32.197 INFO:tasks.rados.rados.0.smithi059.stdout:193: oid 46 version 15 is already newer than 11 2024-04-19T02:03:32.197 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 46 v 15 (ObjNum 97 snap 10 seq_num 97) dirty exists 2024-04-19T02:03:32.197 INFO:tasks.rados.rados.0.smithi059.stdout:193: left oid 46 (ObjNum 97 snap 10 seq_num 97) 2024-04-19T02:03:32.253 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:32.250+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:32.312 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:32.312 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:32.325 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-19T02:03:32.371 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:32.370+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:32.454 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:32.454+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:32.483 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:32.483 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:32.497 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-19T02:03:32.555 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:32.554+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:32.621 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:32.621 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:32.634 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-19T02:03:32.655 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:32.654+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:32.752 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:32.753 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:32.756 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:32.758+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:32.766 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-19T02:03:32.857 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:32.854+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:32.885 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:32.885 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:32.898 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-19T02:03:32.958 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:32.958+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:33.030 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:33.031 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:33.039 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.6 to 1 (10000) 2024-04-19T02:03:33.039 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is set 2024-04-19T02:03:33.046 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-19T02:03:33.046 INFO:tasks.rados.rados.0.smithi059.stdout:192: done (3 left) 2024-04-19T02:03:33.047 INFO:tasks.rados.rados.0.smithi059.stdout:193: done (2 left) 2024-04-19T02:03:33.047 INFO:tasks.rados.rados.0.smithi059.stdout:194: done (1 left) 2024-04-19T02:03:33.047 INFO:tasks.rados.rados.0.smithi059.stdout:195: done (0 left) 2024-04-19T02:03:33.047 INFO:tasks.rados.rados.0.smithi059.stdout:196: copy_from oid 40 from oid 18 current snap is 11 2024-04-19T02:03:33.047 INFO:tasks.rados.rados.0.smithi059.stdout:197: snap_create 2024-04-19T02:03:33.058 INFO:tasks.rados.rados.0.smithi059.stdout:196: finishing copy_from racing read to smithi05930959-40 2024-04-19T02:03:33.065 INFO:tasks.rados.rados.0.smithi059.stdout:196: finishing copy_from to smithi05930959-40 2024-04-19T02:03:33.065 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 40 v 31 (ObjNum 71 snap 5 seq_num 71) dirty exists 2024-04-19T02:03:33.071 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-04-19T02:03:33.098 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:33.098+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:33.160 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:33.158+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:33.166 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:33.167 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:33.180 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-19T02:03:33.266 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:33.266+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:33.345 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:33.345 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:33.359 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-19T02:03:33.362 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:33.358+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:33.463 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:33.462+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:33.528 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:33.528 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:33.543 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-19T02:03:33.586 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:03:33.586+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:03:33.662 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:33.662 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:33.665 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:33.662+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:33.673 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-19T02:03:33.766 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:03:33.762+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:03:33.835 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:33.835 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:33.848 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-19T02:03:33.867 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:33.870+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:33.967 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:33.966+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:34.013 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:34.013 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:34.027 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-19T02:03:34.041 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.7 to 1 (10000) 2024-04-19T02:03:34.047 INFO:tasks.rados.rados.0.smithi059.stdout:196: done (1 left) 2024-04-19T02:03:34.047 INFO:tasks.rados.rados.0.smithi059.stdout:197: done (0 left) 2024-04-19T02:03:34.047 INFO:tasks.rados.rados.0.smithi059.stdout:198: copy_from oid 38 from oid 8 current snap is 12 2024-04-19T02:03:34.047 INFO:tasks.rados.rados.0.smithi059.stdout:199: delete oid 42 current snap is 12 2024-04-19T02:03:34.058 INFO:tasks.rados.rados.0.smithi059.stdout:199: done (1 left) 2024-04-19T02:03:34.059 INFO:tasks.rados.rados.0.smithi059.stdout:200: copy_from oid 4 from oid 16 current snap is 12 2024-04-19T02:03:34.059 INFO:tasks.rados.rados.0.smithi059.stdout:201: rollback oid 12 current snap is 12 2024-04-19T02:03:34.059 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 12 to 11 2024-04-19T02:03:34.059 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 21 current snap is 12 2024-04-19T02:03:34.059 INFO:tasks.rados.rados.0.smithi059.stdout:202: seq_num 99 ranges {0=507904,507904=606208,1114112=557056} 2024-04-19T02:03:34.063 INFO:tasks.rados.rados.0.smithi059.stdout:202: writing smithi05930959-21 from 0 to 507904 tid 1 2024-04-19T02:03:34.068 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:34.066+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:34.070 INFO:tasks.rados.rados.0.smithi059.stdout:202: writing smithi05930959-21 from 507904 to 1114112 tid 2 2024-04-19T02:03:34.076 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-04-19T02:03:34.077 INFO:tasks.rados.rados.0.smithi059.stdout:202: writing smithi05930959-21 from 1114112 to 1671168 tid 3 2024-04-19T02:03:34.077 INFO:tasks.rados.rados.0.smithi059.stdout:203: delete oid 23 current snap is 12 2024-04-19T02:03:34.077 INFO:tasks.rados.rados.0.smithi059.stdout:201: finishing rollback tid 1 to smithi05930959-12 2024-04-19T02:03:34.077 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 12 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-04-19T02:03:34.077 INFO:tasks.rados.rados.0.smithi059.stdout:198: finishing copy_from racing read to smithi05930959-38 2024-04-19T02:03:34.077 INFO:tasks.rados.rados.0.smithi059.stdout:198: finishing copy_from to smithi05930959-38 2024-04-19T02:03:34.078 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 38 v 15 (ObjNum 57 snap 1 seq_num 57) dirty exists 2024-04-19T02:03:34.078 INFO:tasks.rados.rados.0.smithi059.stdout:202: finishing write tid 1 to smithi05930959-21 2024-04-19T02:03:34.085 INFO:tasks.rados.rados.0.smithi059.stdout:202: finishing write tid 2 to smithi05930959-21 2024-04-19T02:03:34.086 INFO:tasks.rados.rados.0.smithi059.stdout:202: finishing write tid 3 to smithi05930959-21 2024-04-19T02:03:34.087 INFO:tasks.rados.rados.0.smithi059.stdout:202: finishing write tid 4 to smithi05930959-21 2024-04-19T02:03:34.087 INFO:tasks.rados.rados.0.smithi059.stdout:198: done (4 left) 2024-04-19T02:03:34.088 INFO:tasks.rados.rados.0.smithi059.stdout:201: done (3 left) 2024-04-19T02:03:34.088 INFO:tasks.rados.rados.0.smithi059.stdout:203: done (2 left) 2024-04-19T02:03:34.088 INFO:tasks.rados.rados.0.smithi059.stdout:204: snap_remove snap 11 2024-04-19T02:03:34.131 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T02:03:34.157 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:34.157 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:34.169 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:34.166+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:34.172 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-19T02:03:34.270 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:34.270+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:34.278 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-04-19T02:03:34.278 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T02:03:34.294 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-04-19T02:03:34.295 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T02:03:34.311 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-04-19T02:03:34.311 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T02:03:34.327 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-04-19T02:03:34.327 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T02:03:34.345 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:34.345 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:34.348 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-04-19T02:03:34.348 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T02:03:34.360 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-19T02:03:34.366 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-04-19T02:03:34.367 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T02:03:34.371 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:34.370+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:34.384 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-04-19T02:03:34.384 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T02:03:34.401 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-04-19T02:03:34.401 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T02:03:34.421 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-04-19T02:03:34.422 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T02:03:34.443 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-04-19T02:03:34.443 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T02:03:34.464 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-04-19T02:03:34.464 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T02:03:34.472 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:34.470+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:34.480 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:34.480 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:34.480 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 0 2024-04-19T02:03:34.480 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:34.485 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-04-19T02:03:34.486 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T02:03:34.495 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-19T02:03:34.573 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:34.570+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:34.659 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:34.659 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:34.659 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-04-19T02:03:34.660 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-04-19T02:03:34.660 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:34.673 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-19T02:03:34.673 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:34.670+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:34.774 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:34.774+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:34.840 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:34.840 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-04-19T02:03:34.840 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-04-19T02:03:34.840 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-04-19T02:03:34.840 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.840 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:351 3.es0 3:7af77758:::smithi05930959-34:head [append 0~622592] snapc 8=[8,6,5] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:34.840 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:24.760252+0000", 2024-04-19T02:03:34.840 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 10.079277241, 2024-04-19T02:03:34.840 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013544021999999999, 2024-04-19T02:03:34.840 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:34.841 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:34.841 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:34.841 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:34.841 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:34.841 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:34.841 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 351 2024-04-19T02:03:34.841 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.841 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:34.841 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.841 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:34.841 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.760252+0000", 2024-04-19T02:03:34.841 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.841 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.841 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.842 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:34.842 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.760252+0000", 2024-04-19T02:03:34.842 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.842 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.842 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.842 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:34.842 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.760255+0000", 2024-04-19T02:03:34.842 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.537e-06 2024-04-19T02:03:34.842 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.842 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.842 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:34.842 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.761579+0000", 2024-04-19T02:03:34.842 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0013242519999999999 2024-04-19T02:03:34.842 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.842 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.843 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:34.843 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.761611+0000", 2024-04-19T02:03:34.843 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1823999999999999e-05 2024-04-19T02:03:34.843 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.843 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.843 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:34.843 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.761637+0000", 2024-04-19T02:03:34.843 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6410999999999998e-05 2024-04-19T02:03:34.843 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.843 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.843 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:34.843 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.761780+0000", 2024-04-19T02:03:34.843 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014312 2024-04-19T02:03:34.843 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.843 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.844 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:34.844 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.761983+0000", 2024-04-19T02:03:34.844 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00020247599999999999 2024-04-19T02:03:34.844 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.844 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.844 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:34.844 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.765031+0000", 2024-04-19T02:03:34.844 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0030481140000000002 2024-04-19T02:03:34.844 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.844 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.844 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:34.844 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.768154+0000", 2024-04-19T02:03:34.844 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003123743 2024-04-19T02:03:34.844 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.844 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.845 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:34.845 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.773547+0000", 2024-04-19T02:03:34.845 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0053928099999999996 2024-04-19T02:03:34.845 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.845 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.845 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:34.845 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.773796+0000", 2024-04-19T02:03:34.845 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00024873499999999998 2024-04-19T02:03:34.845 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.845 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:34.845 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.845 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.845 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.845 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:352 3.es0 3:7af77758:::smithi05930959-34:head [append 0~475136] snapc 8=[8,6,5] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:34.845 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:24.767397+0000", 2024-04-19T02:03:34.846 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 10.072132383, 2024-04-19T02:03:34.846 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013273591, 2024-04-19T02:03:34.846 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:34.846 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:34.846 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:34.846 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:34.849 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:34.849 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:34.849 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 352 2024-04-19T02:03:34.850 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.850 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:34.850 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.850 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:34.850 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.767397+0000", 2024-04-19T02:03:34.850 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.850 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.850 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.850 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:34.850 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.767397+0000", 2024-04-19T02:03:34.850 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.850 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.850 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.850 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:34.850 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.767400+0000", 2024-04-19T02:03:34.851 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1530000000000001e-06 2024-04-19T02:03:34.851 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.851 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.851 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:34.851 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.768322+0000", 2024-04-19T02:03:34.851 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00092243399999999997 2024-04-19T02:03:34.851 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.851 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.851 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:34.851 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.768363+0000", 2024-04-19T02:03:34.851 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0657000000000001e-05 2024-04-19T02:03:34.851 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.851 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.851 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:34.851 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.768388+0000", 2024-04-19T02:03:34.852 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5128000000000001e-05 2024-04-19T02:03:34.852 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.852 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.852 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:34.852 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.768539+0000", 2024-04-19T02:03:34.852 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000150376 2024-04-19T02:03:34.852 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.852 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.852 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:34.852 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.768820+0000", 2024-04-19T02:03:34.852 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00028157699999999998 2024-04-19T02:03:34.852 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.852 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.852 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:34.852 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.771375+0000", 2024-04-19T02:03:34.852 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0025545730000000001 2024-04-19T02:03:34.853 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.853 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.853 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:34.853 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.779788+0000", 2024-04-19T02:03:34.853 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0084135930000000005 2024-04-19T02:03:34.853 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.853 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.853 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:34.853 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.780478+0000", 2024-04-19T02:03:34.853 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00068924100000000005 2024-04-19T02:03:34.853 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.853 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.853 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:34.853 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.780670+0000", 2024-04-19T02:03:34.853 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000192859 2024-04-19T02:03:34.854 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.854 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:34.854 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.854 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.854 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.854 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:355 3.es0 3:7af77758:::smithi05930959-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:34.854 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:24.772697+0000", 2024-04-19T02:03:34.854 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 10.06683198, 2024-04-19T02:03:34.854 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014504697, 2024-04-19T02:03:34.854 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:34.854 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:34.854 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:03:34.854 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:34.854 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:34.854 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:34.855 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 355 2024-04-19T02:03:34.855 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.855 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:34.855 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.855 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:34.855 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.772697+0000", 2024-04-19T02:03:34.855 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.855 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.855 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.855 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:34.855 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.772697+0000", 2024-04-19T02:03:34.855 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.855 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.855 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.856 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:34.856 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.772699+0000", 2024-04-19T02:03:34.856 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3430000000000001e-06 2024-04-19T02:03:34.856 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.856 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.856 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:34.856 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.772706+0000", 2024-04-19T02:03:34.856 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2400000000000001e-06 2024-04-19T02:03:34.856 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.856 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.856 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:34.856 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.772724+0000", 2024-04-19T02:03:34.856 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8289000000000001e-05 2024-04-19T02:03:34.856 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.856 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.856 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:34.857 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.772738+0000", 2024-04-19T02:03:34.857 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3732e-05 2024-04-19T02:03:34.857 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.857 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.857 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:34.857 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.784149+0000", 2024-04-19T02:03:34.857 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011411361 2024-04-19T02:03:34.857 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.857 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.857 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:34.857 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.784288+0000", 2024-04-19T02:03:34.857 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013880699999999999 2024-04-19T02:03:34.857 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.857 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.857 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:34.858 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.787202+0000", 2024-04-19T02:03:34.858 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0029139249999999999 2024-04-19T02:03:34.858 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.858 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:34.858 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.858 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.858 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.858 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.bs7 42/26 ECSubWrite(tid=133, reqid=osd.3.0:133, at_version=42'21, trim_to=0'0, roll_forward_to=39'19))", 2024-04-19T02:03:34.858 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.748986+0000", 2024-04-19T02:03:34.858 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 8.0905428100000005, 2024-04-19T02:03:34.858 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.015850517000000001, 2024-04-19T02:03:34.858 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:34.858 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:34.858 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:34.859 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:34.859 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.859 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:34.859 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748986+0000", 2024-04-19T02:03:34.859 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.859 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.859 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.859 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:34.859 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748986+0000", 2024-04-19T02:03:34.859 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.859 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.859 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.859 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:34.859 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748987+0000", 2024-04-19T02:03:34.859 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6000000000000003e-07 2024-04-19T02:03:34.860 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.860 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.860 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:34.860 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749579+0000", 2024-04-19T02:03:34.860 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00059223999999999995 2024-04-19T02:03:34.860 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.860 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.860 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:34.860 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749611+0000", 2024-04-19T02:03:34.860 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1554999999999999e-05 2024-04-19T02:03:34.860 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.860 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.860 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:34.860 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749633+0000", 2024-04-19T02:03:34.861 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1885000000000001e-05 2024-04-19T02:03:34.861 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.861 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.861 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:34.861 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749799+0000", 2024-04-19T02:03:34.861 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000166422 2024-04-19T02:03:34.861 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.861 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.861 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:34.861 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749825+0000", 2024-04-19T02:03:34.861 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5825000000000001e-05 2024-04-19T02:03:34.861 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.861 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.861 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:34.861 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.764686+0000", 2024-04-19T02:03:34.861 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014861087 2024-04-19T02:03:34.862 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.862 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.862 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:34.862 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.764837+0000", 2024-04-19T02:03:34.862 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000150743 2024-04-19T02:03:34.862 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.862 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:34.862 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.862 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.862 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.862 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.bs7 42/26 ECSubWrite(tid=135, reqid=osd.3.0:135, at_version=42'23, trim_to=0'0, roll_forward_to=39'19))", 2024-04-19T02:03:34.862 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.749928+0000", 2024-04-19T02:03:34.862 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 8.0896014259999998, 2024-04-19T02:03:34.862 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.017644765, 2024-04-19T02:03:34.862 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:34.863 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:34.863 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:34.863 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:34.863 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.863 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:34.863 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749928+0000", 2024-04-19T02:03:34.863 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.863 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.863 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.863 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:34.863 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749928+0000", 2024-04-19T02:03:34.863 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.863 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.863 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.863 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:34.864 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749929+0000", 2024-04-19T02:03:34.864 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.136e-06 2024-04-19T02:03:34.864 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.864 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.864 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:34.864 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749999+0000", 2024-04-19T02:03:34.864 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0471999999999999e-05 2024-04-19T02:03:34.864 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.864 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.864 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:34.864 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.750024+0000", 2024-04-19T02:03:34.864 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4575000000000001e-05 2024-04-19T02:03:34.864 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.864 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.864 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:34.865 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.750044+0000", 2024-04-19T02:03:34.865 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9802000000000001e-05 2024-04-19T02:03:34.865 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.865 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.865 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:34.865 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.756068+0000", 2024-04-19T02:03:34.865 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0060238050000000001 2024-04-19T02:03:34.865 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.865 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.865 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:34.865 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.756097+0000", 2024-04-19T02:03:34.865 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9045000000000001e-05 2024-04-19T02:03:34.865 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.865 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.865 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:34.865 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.767487+0000", 2024-04-19T02:03:34.866 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011390612 2024-04-19T02:03:34.866 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.866 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.866 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:34.866 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.767573+0000", 2024-04-19T02:03:34.866 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5317999999999999e-05 2024-04-19T02:03:34.866 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.866 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:34.866 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.866 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.866 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.866 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.4s4 42/26 ECSubWrite(tid=45, reqid=osd.1.0:44, at_version=0'0, trim_to=0'0, roll_forward_to=42'32))", 2024-04-19T02:03:34.866 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.755829+0000", 2024-04-19T02:03:34.866 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 8.0837003559999996, 2024-04-19T02:03:34.866 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013143623, 2024-04-19T02:03:34.867 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:34.867 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:34.867 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:34.867 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:34.867 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.867 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:34.867 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755829+0000", 2024-04-19T02:03:34.867 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.867 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.867 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.867 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:34.867 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755829+0000", 2024-04-19T02:03:34.867 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.867 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.867 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.868 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:34.868 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755831+0000", 2024-04-19T02:03:34.868 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6929999999999999e-06 2024-04-19T02:03:34.868 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.868 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.868 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:34.868 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755858+0000", 2024-04-19T02:03:34.868 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7067999999999999e-05 2024-04-19T02:03:34.868 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.868 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.868 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:34.868 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755887+0000", 2024-04-19T02:03:34.868 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9651000000000001e-05 2024-04-19T02:03:34.868 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.868 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.869 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:34.869 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755910+0000", 2024-04-19T02:03:34.869 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2311999999999999e-05 2024-04-19T02:03:34.869 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.869 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.869 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:34.869 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.756066+0000", 2024-04-19T02:03:34.869 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000156176 2024-04-19T02:03:34.869 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.869 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.869 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:34.869 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.756092+0000", 2024-04-19T02:03:34.869 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5825000000000001e-05 2024-04-19T02:03:34.869 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.869 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.870 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:34.870 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.768920+0000", 2024-04-19T02:03:34.870 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012828104 2024-04-19T02:03:34.870 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.870 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.870 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:34.870 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.768972+0000", 2024-04-19T02:03:34.870 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2794000000000003e-05 2024-04-19T02:03:34.870 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.870 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:34.870 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.870 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.870 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.870 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.ds2 46/26 ECSubWrite(tid=187, reqid=osd.8.0:187, at_version=46'35, trim_to=0'0, roll_forward_to=45'31))", 2024-04-19T02:03:34.871 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.058580+0000", 2024-04-19T02:03:34.871 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.7809493280000002, 2024-04-19T02:03:34.871 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.016482007999999999, 2024-04-19T02:03:34.871 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:34.871 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:34.871 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:34.871 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:34.871 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.871 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:34.871 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.058580+0000", 2024-04-19T02:03:34.871 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.871 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.871 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.871 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:34.871 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.058580+0000", 2024-04-19T02:03:34.872 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.872 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.872 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.872 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:34.872 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.058582+0000", 2024-04-19T02:03:34.872 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7069999999999999e-06 2024-04-19T02:03:34.872 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.872 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.872 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:34.872 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.058632+0000", 2024-04-19T02:03:34.872 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0710000000000001e-05 2024-04-19T02:03:34.872 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.872 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.872 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:34.873 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.058664+0000", 2024-04-19T02:03:34.873 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1448000000000001e-05 2024-04-19T02:03:34.873 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.873 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.873 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:34.873 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.058688+0000", 2024-04-19T02:03:34.873 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4405000000000001e-05 2024-04-19T02:03:34.873 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.873 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.873 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:34.873 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.059171+0000", 2024-04-19T02:03:34.873 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00048236699999999998 2024-04-19T02:03:34.873 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.873 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.873 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:34.874 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.059195+0000", 2024-04-19T02:03:34.874 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4042000000000001e-05 2024-04-19T02:03:34.874 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.874 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.874 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:34.874 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.074940+0000", 2024-04-19T02:03:34.874 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.015745333 2024-04-19T02:03:34.874 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.874 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.874 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:34.874 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.075062+0000", 2024-04-19T02:03:34.874 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000121996 2024-04-19T02:03:34.874 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.874 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:34.874 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.875 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.875 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.875 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.4s4 46/26 ECSubWrite(tid=58, reqid=osd.1.0:58, at_version=46'44, trim_to=0'0, roll_forward_to=45'42))", 2024-04-19T02:03:34.875 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.059794+0000", 2024-04-19T02:03:34.875 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.779735649, 2024-04-19T02:03:34.875 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.015185597, 2024-04-19T02:03:34.875 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:34.875 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:34.875 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:34.875 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:34.875 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.875 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:34.875 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.059794+0000", 2024-04-19T02:03:34.875 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.875 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.876 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.876 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:34.876 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.059794+0000", 2024-04-19T02:03:34.876 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.876 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.876 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.876 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:34.876 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.059794+0000", 2024-04-19T02:03:34.876 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2399999999999998e-07 2024-04-19T02:03:34.876 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.876 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.876 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:34.876 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.059843+0000", 2024-04-19T02:03:34.876 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9217e-05 2024-04-19T02:03:34.876 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.876 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.877 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:34.877 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.059888+0000", 2024-04-19T02:03:34.877 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4842999999999997e-05 2024-04-19T02:03:34.877 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.877 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.877 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:34.877 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.059911+0000", 2024-04-19T02:03:34.877 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2829000000000001e-05 2024-04-19T02:03:34.877 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.877 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.877 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:34.877 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.060084+0000", 2024-04-19T02:03:34.877 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017278000000000001 2024-04-19T02:03:34.877 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.878 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.878 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:34.878 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.060110+0000", 2024-04-19T02:03:34.878 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5931999999999998e-05 2024-04-19T02:03:34.878 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.878 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.878 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:34.878 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.074866+0000", 2024-04-19T02:03:34.878 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014756136 2024-04-19T02:03:34.878 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.878 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.878 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:34.878 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.074979+0000", 2024-04-19T02:03:34.878 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000113236 2024-04-19T02:03:34.878 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.879 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:34.879 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.879 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.879 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.879 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.as5 46/26 ECSubWrite(tid=102, reqid=osd.11.0:102, at_version=46'47, trim_to=0'0, roll_forward_to=45'45))", 2024-04-19T02:03:34.879 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.066484+0000", 2024-04-19T02:03:34.879 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.7730456649999997, 2024-04-19T02:03:34.879 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014065139000000001, 2024-04-19T02:03:34.879 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:34.879 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:34.879 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:34.879 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:34.879 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.879 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:34.879 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066484+0000", 2024-04-19T02:03:34.880 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.880 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.880 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.880 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:34.880 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066484+0000", 2024-04-19T02:03:34.880 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.880 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.880 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.880 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:34.880 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066484+0000", 2024-04-19T02:03:34.880 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0299999999999998e-07 2024-04-19T02:03:34.880 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.880 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.880 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:34.881 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066542+0000", 2024-04-19T02:03:34.881 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7182999999999998e-05 2024-04-19T02:03:34.881 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.881 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.881 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:34.881 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066572+0000", 2024-04-19T02:03:34.881 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0205e-05 2024-04-19T02:03:34.881 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.881 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.881 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:34.881 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066594+0000", 2024-04-19T02:03:34.881 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2478999999999999e-05 2024-04-19T02:03:34.881 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.881 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.881 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:34.882 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066759+0000", 2024-04-19T02:03:34.882 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016516999999999999 2024-04-19T02:03:34.882 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.882 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.882 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:34.882 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066784+0000", 2024-04-19T02:03:34.882 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4587999999999999e-05 2024-04-19T02:03:34.882 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.882 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.882 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:34.882 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.080462+0000", 2024-04-19T02:03:34.882 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013678459 2024-04-19T02:03:34.882 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.882 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.883 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:34.883 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.080549+0000", 2024-04-19T02:03:34.883 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.6252e-05 2024-04-19T02:03:34.883 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.883 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:34.883 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.883 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.883 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.883 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.cs3 46/26 ECSubWrite(tid=44, reqid=osd.9.0:44, at_version=46'26, trim_to=0'0, roll_forward_to=45'24))", 2024-04-19T02:03:34.883 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.066755+0000", 2024-04-19T02:03:34.883 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.7727738979999996, 2024-04-19T02:03:34.883 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.01420738, 2024-04-19T02:03:34.883 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:34.883 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:34.884 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:34.884 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:34.884 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.884 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:34.884 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066755+0000", 2024-04-19T02:03:34.884 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.884 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.884 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.884 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:34.884 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066755+0000", 2024-04-19T02:03:34.884 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.884 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.884 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.884 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:34.884 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066756+0000", 2024-04-19T02:03:34.885 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4000000000000002e-07 2024-04-19T02:03:34.885 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.885 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.885 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:34.885 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066799+0000", 2024-04-19T02:03:34.885 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3217000000000003e-05 2024-04-19T02:03:34.885 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.885 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.885 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:34.885 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066825+0000", 2024-04-19T02:03:34.885 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6248999999999999e-05 2024-04-19T02:03:34.885 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.885 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.885 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:34.886 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066851+0000", 2024-04-19T02:03:34.886 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5171e-05 2024-04-19T02:03:34.886 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.886 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.886 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:34.886 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.067016+0000", 2024-04-19T02:03:34.886 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016518699999999999 2024-04-19T02:03:34.886 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.886 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.886 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:34.886 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.067042+0000", 2024-04-19T02:03:34.886 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5992000000000001e-05 2024-04-19T02:03:34.886 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.886 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.886 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:34.887 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.080884+0000", 2024-04-19T02:03:34.887 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013842047999999999 2024-04-19T02:03:34.887 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.887 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.887 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:34.887 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.080963+0000", 2024-04-19T02:03:34.887 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8975999999999997e-05 2024-04-19T02:03:34.887 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.887 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:34.887 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.887 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.887 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.887 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.as5 46/26 ECSubWrite(tid=103, reqid=osd.11.0:103, at_version=46'49, trim_to=0'0, roll_forward_to=45'45))", 2024-04-19T02:03:34.887 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.067427+0000", 2024-04-19T02:03:34.888 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.7721017329999995, 2024-04-19T02:03:34.888 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.020157152000000001, 2024-04-19T02:03:34.888 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:34.888 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:34.888 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:34.888 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:34.888 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.888 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:34.888 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.067427+0000", 2024-04-19T02:03:34.888 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.888 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.888 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.888 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:34.888 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.067427+0000", 2024-04-19T02:03:34.888 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.888 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.889 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.889 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:34.889 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.067429+0000", 2024-04-19T02:03:34.889 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7439999999999999e-06 2024-04-19T02:03:34.889 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.889 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.889 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:34.889 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.067489+0000", 2024-04-19T02:03:34.889 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9988999999999998e-05 2024-04-19T02:03:34.889 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.889 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.889 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:34.889 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.068116+0000", 2024-04-19T02:03:34.889 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00062675199999999995 2024-04-19T02:03:34.889 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.889 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.890 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:34.890 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.068146+0000", 2024-04-19T02:03:34.890 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9751999999999999e-05 2024-04-19T02:03:34.890 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.890 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.890 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:34.890 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.070935+0000", 2024-04-19T02:03:34.890 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0027894399999999998 2024-04-19T02:03:34.890 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.890 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.890 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:34.890 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.070950+0000", 2024-04-19T02:03:34.890 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5218999999999999e-05 2024-04-19T02:03:34.890 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.890 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.891 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:34.891 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.087492+0000", 2024-04-19T02:03:34.891 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.016541891999999999 2024-04-19T02:03:34.891 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.891 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.891 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:34.891 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.087585+0000", 2024-04-19T02:03:34.891 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.2363999999999999e-05 2024-04-19T02:03:34.891 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.891 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:34.891 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.891 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.891 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.891 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.7s4 46/26 ECSubWrite(tid=158, reqid=osd.3.0:158, at_version=46'14, trim_to=0'0, roll_forward_to=45'12))", 2024-04-19T02:03:34.891 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.072853+0000", 2024-04-19T02:03:34.892 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.7666757359999998, 2024-04-19T02:03:34.892 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014228316, 2024-04-19T02:03:34.892 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:34.892 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:34.892 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:34.892 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:34.892 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.892 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:34.892 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072853+0000", 2024-04-19T02:03:34.892 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.892 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.892 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.892 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:34.892 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072853+0000", 2024-04-19T02:03:34.893 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.893 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.893 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.893 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:34.893 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072854+0000", 2024-04-19T02:03:34.893 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5700000000000002e-07 2024-04-19T02:03:34.893 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.893 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.893 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:34.893 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072905+0000", 2024-04-19T02:03:34.893 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1443000000000002e-05 2024-04-19T02:03:34.893 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.893 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.893 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:34.893 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072931+0000", 2024-04-19T02:03:34.894 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5103e-05 2024-04-19T02:03:34.894 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.894 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.894 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:34.894 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072953+0000", 2024-04-19T02:03:34.894 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2174000000000001e-05 2024-04-19T02:03:34.894 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.894 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.894 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:34.894 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.073226+0000", 2024-04-19T02:03:34.894 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000273174 2024-04-19T02:03:34.894 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.894 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.894 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:34.895 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.073252+0000", 2024-04-19T02:03:34.895 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6152e-05 2024-04-19T02:03:34.895 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.895 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.895 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:34.895 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.086987+0000", 2024-04-19T02:03:34.895 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013734837999999999 2024-04-19T02:03:34.895 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.895 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.895 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:34.895 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.087082+0000", 2024-04-19T02:03:34.895 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4875000000000002e-05 2024-04-19T02:03:34.895 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.895 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:34.895 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.896 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.896 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.896 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.4s4 46/26 ECSubWrite(tid=59, reqid=osd.1.0:58, at_version=0'0, trim_to=0'0, roll_forward_to=46'44))", 2024-04-19T02:03:34.896 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.076912+0000", 2024-04-19T02:03:34.896 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.7626168150000003, 2024-04-19T02:03:34.896 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014831941, 2024-04-19T02:03:34.896 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:34.896 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:34.896 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:34.896 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:34.896 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.896 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:34.896 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.076912+0000", 2024-04-19T02:03:34.896 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.897 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.897 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.897 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:34.897 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.076912+0000", 2024-04-19T02:03:34.897 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.897 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.897 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.897 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:34.897 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.076914+0000", 2024-04-19T02:03:34.897 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.829e-06 2024-04-19T02:03:34.897 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.897 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.897 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:34.897 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.076944+0000", 2024-04-19T02:03:34.897 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9482e-05 2024-04-19T02:03:34.897 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.898 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.898 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:34.898 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.080072+0000", 2024-04-19T02:03:34.898 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0031280629999999999 2024-04-19T02:03:34.898 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.898 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.898 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:34.898 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.080101+0000", 2024-04-19T02:03:34.898 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9524999999999999e-05 2024-04-19T02:03:34.898 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.898 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.898 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:34.898 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.081114+0000", 2024-04-19T02:03:34.898 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0010130250000000001 2024-04-19T02:03:34.899 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.899 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.899 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:34.899 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.081138+0000", 2024-04-19T02:03:34.899 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4105e-05 2024-04-19T02:03:34.899 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.899 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.899 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:34.899 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.091652+0000", 2024-04-19T02:03:34.899 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010514055 2024-04-19T02:03:34.899 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.899 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.899 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:34.899 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.091744+0000", 2024-04-19T02:03:34.899 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1856999999999998e-05 2024-04-19T02:03:34.899 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.900 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:34.900 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.900 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.900 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.900 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.1s5 46/26 ECSubWrite(tid=21, reqid=client.4509.0:416, at_version=46'9, trim_to=0'0, roll_forward_to=29'6))", 2024-04-19T02:03:34.900 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.081553+0000", 2024-04-19T02:03:34.900 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.7579764710000001, 2024-04-19T02:03:34.900 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013737042, 2024-04-19T02:03:34.900 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:34.900 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:34.900 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:34.900 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:34.900 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.900 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:34.901 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.081553+0000", 2024-04-19T02:03:34.901 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.901 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.901 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.901 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:34.901 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.081553+0000", 2024-04-19T02:03:34.901 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.901 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.901 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.901 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:34.901 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.081554+0000", 2024-04-19T02:03:34.901 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.083e-06 2024-04-19T02:03:34.901 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.901 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.901 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:34.901 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.081793+0000", 2024-04-19T02:03:34.902 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00023883199999999999 2024-04-19T02:03:34.902 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.902 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.902 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:34.902 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.081820+0000", 2024-04-19T02:03:34.902 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7429000000000001e-05 2024-04-19T02:03:34.902 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.902 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.902 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:34.902 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.081840+0000", 2024-04-19T02:03:34.902 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9508e-05 2024-04-19T02:03:34.902 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.902 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.902 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:34.903 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.085041+0000", 2024-04-19T02:03:34.903 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0032012859999999998 2024-04-19T02:03:34.903 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.903 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.903 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:34.903 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.085066+0000", 2024-04-19T02:03:34.903 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5222e-05 2024-04-19T02:03:34.903 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.903 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.903 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:34.903 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.095186+0000", 2024-04-19T02:03:34.903 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010120168000000001 2024-04-19T02:03:34.903 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.903 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.903 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:34.903 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.095290+0000", 2024-04-19T02:03:34.904 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000103514 2024-04-19T02:03:34.904 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.904 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:34.904 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.904 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.904 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.904 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:420 3.es0 3:7831c41d:::smithi05930959-30:head [copy-from ver 14] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:34.904 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.193498+0000", 2024-04-19T02:03:34.904 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.6460311389999998, 2024-04-19T02:03:34.904 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.052261775000000003, 2024-04-19T02:03:34.904 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:34.904 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:34.904 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:34.904 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:34.904 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:34.905 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:34.905 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 420 2024-04-19T02:03:34.905 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.905 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:34.905 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.905 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:34.905 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193498+0000", 2024-04-19T02:03:34.905 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.905 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.905 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.905 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:34.905 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193498+0000", 2024-04-19T02:03:34.905 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.905 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.905 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.905 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:34.906 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193501+0000", 2024-04-19T02:03:34.906 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.88e-06 2024-04-19T02:03:34.906 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.906 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.906 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:34.906 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193514+0000", 2024-04-19T02:03:34.906 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2879e-05 2024-04-19T02:03:34.906 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.906 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.906 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:34.906 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193546+0000", 2024-04-19T02:03:34.906 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1931999999999999e-05 2024-04-19T02:03:34.906 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.906 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.906 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:34.906 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193560+0000", 2024-04-19T02:03:34.907 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4609e-05 2024-04-19T02:03:34.907 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.907 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.907 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:34.907 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193695+0000", 2024-04-19T02:03:34.907 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013434499999999999 2024-04-19T02:03:34.907 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.907 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.907 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:34.907 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193864+0000", 2024-04-19T02:03:34.907 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016894300000000001 2024-04-19T02:03:34.907 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.907 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.907 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:34.907 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.239884+0000", 2024-04-19T02:03:34.908 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.046020559000000003 2024-04-19T02:03:34.908 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.908 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.908 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:34.908 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.245339+0000", 2024-04-19T02:03:34.908 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0054546569999999999 2024-04-19T02:03:34.908 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.908 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.908 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:34.908 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.245619+0000", 2024-04-19T02:03:34.908 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00028016800000000002 2024-04-19T02:03:34.908 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.908 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.908 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:34.908 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.245760+0000", 2024-04-19T02:03:34.908 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000140803 2024-04-19T02:03:34.909 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.909 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:34.909 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.909 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.909 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.909 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:421 3.es0 3:7831c41d:::smithi05930959-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:34.909 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.193593+0000", 2024-04-19T02:03:34.909 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.645936635, 2024-04-19T02:03:34.909 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.049216688000000001, 2024-04-19T02:03:34.909 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:34.909 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:34.909 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:03:34.909 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:34.909 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:34.909 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:34.910 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 421 2024-04-19T02:03:34.910 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.910 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:34.910 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.910 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:34.910 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193593+0000", 2024-04-19T02:03:34.910 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.910 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.910 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.910 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:34.910 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193593+0000", 2024-04-19T02:03:34.910 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.910 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.910 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.910 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:34.911 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193594+0000", 2024-04-19T02:03:34.911 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4169999999999999e-06 2024-04-19T02:03:34.911 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.911 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.911 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:34.911 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193601+0000", 2024-04-19T02:03:34.911 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6429999999999998e-06 2024-04-19T02:03:34.911 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.911 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.911 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:34.911 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193621+0000", 2024-04-19T02:03:34.911 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0435e-05 2024-04-19T02:03:34.911 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.911 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.911 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:34.912 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193636+0000", 2024-04-19T02:03:34.912 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4822e-05 2024-04-19T02:03:34.912 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.912 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.912 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:34.912 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.194259+0000", 2024-04-19T02:03:34.912 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00062306299999999998 2024-04-19T02:03:34.912 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.912 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.912 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for blocked object", 2024-04-19T02:03:34.912 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.194433+0000", 2024-04-19T02:03:34.912 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017404899999999999 2024-04-19T02:03:34.912 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.912 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.912 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:34.913 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.242239+0000", 2024-04-19T02:03:34.913 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.047806081 2024-04-19T02:03:34.913 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.913 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.913 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:34.913 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.242500+0000", 2024-04-19T02:03:34.913 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00026063699999999998 2024-04-19T02:03:34.913 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.913 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.913 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:34.913 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.242809+0000", 2024-04-19T02:03:34.913 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00030954099999999998 2024-04-19T02:03:34.913 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.913 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:34.913 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.913 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.914 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.914 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.bs7 47/26 ECSubWrite(tid=183, reqid=osd.3.0:183, at_version=47'38, trim_to=0'0, roll_forward_to=46'34))", 2024-04-19T02:03:34.914 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.084378+0000", 2024-04-19T02:03:34.914 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 3.7551515479999997, 2024-04-19T02:03:34.914 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.020336838999999999, 2024-04-19T02:03:34.914 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:34.914 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:34.914 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:34.914 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:34.914 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.914 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:34.914 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.084378+0000", 2024-04-19T02:03:34.914 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.914 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.914 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.915 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:34.915 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.084378+0000", 2024-04-19T02:03:34.915 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.915 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.915 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.915 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:34.915 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.084379+0000", 2024-04-19T02:03:34.915 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3230000000000001e-06 2024-04-19T02:03:34.915 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.915 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.915 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:34.915 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.084443+0000", 2024-04-19T02:03:34.915 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3745999999999996e-05 2024-04-19T02:03:34.915 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.915 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.916 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:34.916 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.084476+0000", 2024-04-19T02:03:34.916 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3352000000000002e-05 2024-04-19T02:03:34.916 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.916 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.916 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:34.916 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.084500+0000", 2024-04-19T02:03:34.916 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3492000000000001e-05 2024-04-19T02:03:34.916 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.916 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.916 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:34.916 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.084742+0000", 2024-04-19T02:03:34.916 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00024195900000000001 2024-04-19T02:03:34.916 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.916 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.916 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:34.917 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.084769+0000", 2024-04-19T02:03:34.917 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7222000000000001e-05 2024-04-19T02:03:34.917 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.917 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.917 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:34.917 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.104626+0000", 2024-04-19T02:03:34.917 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.019857593999999999 2024-04-19T02:03:34.917 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.917 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.917 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:34.917 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.104715+0000", 2024-04-19T02:03:34.917 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8151000000000004e-05 2024-04-19T02:03:34.917 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.917 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:34.917 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.917 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.918 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.918 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.1s5 47/26 ECSubWrite(tid=30, reqid=osd.0.0:30, at_version=47'17, trim_to=0'0, roll_forward_to=46'13))", 2024-04-19T02:03:34.918 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.085073+0000", 2024-04-19T02:03:34.918 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 3.754456523, 2024-04-19T02:03:34.918 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.019688054999999999, 2024-04-19T02:03:34.918 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:34.918 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:34.918 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:34.918 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:34.918 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.918 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:34.918 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085073+0000", 2024-04-19T02:03:34.918 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.918 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.919 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.919 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:34.919 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085073+0000", 2024-04-19T02:03:34.919 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.919 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.919 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.919 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:34.919 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085075+0000", 2024-04-19T02:03:34.919 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.81e-06 2024-04-19T02:03:34.919 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.919 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.919 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:34.919 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085126+0000", 2024-04-19T02:03:34.919 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1100000000000002e-05 2024-04-19T02:03:34.919 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.920 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.920 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:34.920 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085158+0000", 2024-04-19T02:03:34.920 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.2375000000000001e-05 2024-04-19T02:03:34.920 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.920 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.920 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:34.920 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085180+0000", 2024-04-19T02:03:34.920 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1942000000000001e-05 2024-04-19T02:03:34.920 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.920 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.920 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:34.920 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.086348+0000", 2024-04-19T02:03:34.920 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001167757 2024-04-19T02:03:34.920 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.920 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.921 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:34.921 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.086362+0000", 2024-04-19T02:03:34.921 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4749000000000001e-05 2024-04-19T02:03:34.921 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.921 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.921 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:34.921 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.104692+0000", 2024-04-19T02:03:34.921 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.018329525999999999 2024-04-19T02:03:34.921 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.921 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.921 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:34.921 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.104761+0000", 2024-04-19T02:03:34.921 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8795999999999997e-05 2024-04-19T02:03:34.921 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.921 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:34.922 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.922 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.922 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.922 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.8s2 47/26 ECSubWrite(tid=86, reqid=osd.4.0:86, at_version=47'44, trim_to=0'0, roll_forward_to=46'42))", 2024-04-19T02:03:34.922 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.093211+0000", 2024-04-19T02:03:34.922 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 3.7463184649999999, 2024-04-19T02:03:34.922 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014295323, 2024-04-19T02:03:34.922 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:34.922 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:34.922 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:34.922 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:34.922 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.922 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:34.922 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.093211+0000", 2024-04-19T02:03:34.923 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.923 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.923 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.923 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:34.923 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.093211+0000", 2024-04-19T02:03:34.923 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.923 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.923 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.923 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:34.923 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.093211+0000", 2024-04-19T02:03:34.923 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3900000000000004e-07 2024-04-19T02:03:34.923 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.923 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.923 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:34.923 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.093266+0000", 2024-04-19T02:03:34.923 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4684e-05 2024-04-19T02:03:34.924 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.924 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.924 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:34.924 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.093292+0000", 2024-04-19T02:03:34.924 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6001999999999999e-05 2024-04-19T02:03:34.924 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.924 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.924 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:34.924 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.093308+0000", 2024-04-19T02:03:34.924 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6052000000000001e-05 2024-04-19T02:03:34.924 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.924 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.924 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:34.924 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.093541+0000", 2024-04-19T02:03:34.924 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00023266000000000001 2024-04-19T02:03:34.925 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.925 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.925 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:34.925 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.093573+0000", 2024-04-19T02:03:34.925 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1908e-05 2024-04-19T02:03:34.925 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.925 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.925 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:34.925 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.107421+0000", 2024-04-19T02:03:34.925 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013848035999999999 2024-04-19T02:03:34.925 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.925 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.925 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:34.925 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.107506+0000", 2024-04-19T02:03:34.926 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5341999999999997e-05 2024-04-19T02:03:34.926 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.926 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:34.926 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.926 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.926 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.926 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:460 3.es0 3:7af77758:::smithi05930959-34:head [read 0~1867776,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:03:34.926 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.137733+0000", 2024-04-19T02:03:34.926 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 3.701796458, 2024-04-19T02:03:34.926 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013538458, 2024-04-19T02:03:34.926 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:34.926 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:34.926 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:03:34.926 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:34.926 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:34.927 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:34.927 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 460 2024-04-19T02:03:34.927 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.927 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:34.927 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.927 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:34.927 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.137733+0000", 2024-04-19T02:03:34.927 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.927 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.927 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.927 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:34.927 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.137733+0000", 2024-04-19T02:03:34.927 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:34.927 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.928 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.928 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:34.928 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.137734+0000", 2024-04-19T02:03:34.928 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3200000000000001e-06 2024-04-19T02:03:34.928 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.928 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.928 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:34.928 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.137742+0000", 2024-04-19T02:03:34.928 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.763e-06 2024-04-19T02:03:34.928 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.928 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.928 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:34.928 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.137761+0000", 2024-04-19T02:03:34.928 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9358999999999999e-05 2024-04-19T02:03:34.928 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.929 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.929 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:34.929 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.137774+0000", 2024-04-19T02:03:34.929 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3089e-05 2024-04-19T02:03:34.929 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.929 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.929 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:34.929 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.140110+0000", 2024-04-19T02:03:34.929 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0023359129999999998 2024-04-19T02:03:34.929 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.929 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.930 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:34.930 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.140354+0000", 2024-04-19T02:03:34.930 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00024421600000000001 2024-04-19T02:03:34.930 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:34.930 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:34.930 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:34.930 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.151271+0000", 2024-04-19T02:03:34.930 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010916798 2024-04-19T02:03:34.930 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.930 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:34.930 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.930 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:34.930 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:34.930 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:34.932 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-19T02:03:34.933 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:34.934+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:35.033 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:35.034+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:35.063 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:35.063 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:03:35.063 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-04-19T02:03:35.063 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:35.073 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-19T02:03:35.077 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-19T02:03:35.094 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.8 to 1 (10000) 2024-04-19T02:03:35.102 INFO:tasks.rados.rados.0.smithi059.stdout:204: done (2 left) 2024-04-19T02:03:35.102 INFO:tasks.rados.rados.0.smithi059.stdout:205: snap_create 2024-04-19T02:03:35.102 INFO:tasks.rados.rados.0.smithi059.stdout:202: finishing write tid 5 to smithi05930959-21 2024-04-19T02:03:35.102 INFO:tasks.rados.rados.0.smithi059.stdout:202: oid 21 updating version 0 to 96 2024-04-19T02:03:35.102 INFO:tasks.rados.rados.0.smithi059.stdout:202: oid 21 version 96 is already newer than 94 2024-04-19T02:03:35.102 INFO:tasks.rados.rados.0.smithi059.stdout:202: oid 21 version 96 is already newer than 93 2024-04-19T02:03:35.102 INFO:tasks.rados.rados.0.smithi059.stdout:202: oid 21 version 96 is already newer than 95 2024-04-19T02:03:35.102 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 21 v 96 (ObjNum 99 snap 12 seq_num 99) dirty exists 2024-04-19T02:03:35.102 INFO:tasks.rados.rados.0.smithi059.stdout:202: left oid 21 (ObjNum 99 snap 12 seq_num 99) 2024-04-19T02:03:35.102 INFO:tasks.rados.rados.0.smithi059.stdout:200: finishing copy_from racing read to smithi05930959-4 2024-04-19T02:03:35.103 INFO:tasks.rados.rados.0.smithi059.stdout:200: finishing copy_from to smithi05930959-4 2024-04-19T02:03:35.103 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 4 v 41 (ObjNum 84 snap 8 seq_num 84) dirty exists 2024-04-19T02:03:35.125 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-04-19T02:03:35.171 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:35.170+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:35.235 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:35.234+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:35.237 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:35.238 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:03:35.238 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-04-19T02:03:35.238 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-04-19T02:03:35.238 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:35.251 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-19T02:03:35.336 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:35.334+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:35.380 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:35.380 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-04-19T02:03:35.380 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-04-19T02:03:35.380 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-04-19T02:03:35.381 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.381 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:35.381 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:02:59.539411+0000", 2024-04-19T02:03:35.381 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 35.841185783999997, 2024-04-19T02:03:35.381 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.43600685099999997, 2024-04-19T02:03:35.381 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:35.381 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:35.382 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:35.382 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:35.382 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:35.382 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:35.382 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 1 2024-04-19T02:03:35.382 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.382 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:35.382 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.382 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:35.382 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539411+0000", 2024-04-19T02:03:35.382 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.382 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.383 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.383 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:35.383 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539411+0000", 2024-04-19T02:03:35.383 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.383 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.383 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.383 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:35.383 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539413+0000", 2024-04-19T02:03:35.383 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3970000000000001e-06 2024-04-19T02:03:35.383 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.383 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.384 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:35.384 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539429+0000", 2024-04-19T02:03:35.384 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5153e-05 2024-04-19T02:03:35.384 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.384 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.384 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:35.384 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539450+0000", 2024-04-19T02:03:35.384 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1243e-05 2024-04-19T02:03:35.384 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.384 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.384 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:35.384 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539470+0000", 2024-04-19T02:03:35.384 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0017000000000002e-05 2024-04-19T02:03:35.384 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.384 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.385 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:35.385 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539605+0000", 2024-04-19T02:03:35.385 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013489200000000001 2024-04-19T02:03:35.385 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.385 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.385 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-04-19T02:03:35.385 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539633+0000", 2024-04-19T02:03:35.385 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8113e-05 2024-04-19T02:03:35.385 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.385 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.385 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:35.385 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.972415+0000", 2024-04-19T02:03:35.385 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.43278235300000001 2024-04-19T02:03:35.385 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.385 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.386 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:35.386 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.972942+0000", 2024-04-19T02:03:35.386 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000526379 2024-04-19T02:03:35.386 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.386 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.386 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:35.386 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.973384+0000", 2024-04-19T02:03:35.386 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00044265 2024-04-19T02:03:35.386 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.386 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.386 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:35.387 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.974795+0000", 2024-04-19T02:03:35.387 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001410592 2024-04-19T02:03:35.387 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.387 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.387 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:35.387 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.975249+0000", 2024-04-19T02:03:35.387 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00045376999999999999 2024-04-19T02:03:35.387 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.387 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.387 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:35.387 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.975346+0000", 2024-04-19T02:03:35.387 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7916000000000002e-05 2024-04-19T02:03:35.387 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.387 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.387 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:35.388 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.975418+0000", 2024-04-19T02:03:35.388 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1376000000000002e-05 2024-04-19T02:03:35.388 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.388 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:35.388 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.388 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.388 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.388 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:35.388 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.313928+0000", 2024-04-19T02:03:35.388 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 35.066668708999998, 2024-04-19T02:03:35.388 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017995457999999999, 2024-04-19T02:03:35.388 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:35.388 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:35.388 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:35.388 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:35.389 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:35.389 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:35.389 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 115 2024-04-19T02:03:35.389 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.389 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:35.389 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.389 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:35.389 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313928+0000", 2024-04-19T02:03:35.389 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.390 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.390 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.390 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:35.390 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313928+0000", 2024-04-19T02:03:35.390 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.390 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.390 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.390 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:35.390 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.313929+0000", 2024-04-19T02:03:35.390 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.319e-06 2024-04-19T02:03:35.391 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.391 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.391 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:35.391 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314134+0000", 2024-04-19T02:03:35.391 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020471899999999999 2024-04-19T02:03:35.391 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.391 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.391 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:35.391 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314157+0000", 2024-04-19T02:03:35.391 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.27e-05 2024-04-19T02:03:35.391 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.392 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.392 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:35.392 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314174+0000", 2024-04-19T02:03:35.392 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.694e-05 2024-04-19T02:03:35.392 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.392 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.392 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:35.392 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.325978+0000", 2024-04-19T02:03:35.392 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011804385000000001 2024-04-19T02:03:35.392 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.392 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.392 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:35.392 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.326092+0000", 2024-04-19T02:03:35.392 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000113916 2024-04-19T02:03:35.392 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.393 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.393 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:35.393 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.326279+0000", 2024-04-19T02:03:35.393 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000186632 2024-04-19T02:03:35.393 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.393 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.393 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:35.393 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.328477+0000", 2024-04-19T02:03:35.393 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0021981660000000001 2024-04-19T02:03:35.393 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.393 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.393 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:35.393 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.331786+0000", 2024-04-19T02:03:35.393 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003309055 2024-04-19T02:03:35.393 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.393 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.394 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:35.394 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.331864+0000", 2024-04-19T02:03:35.394 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8382999999999998e-05 2024-04-19T02:03:35.394 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.394 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.394 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:35.394 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.331923+0000", 2024-04-19T02:03:35.394 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9243000000000002e-05 2024-04-19T02:03:35.394 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.394 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:35.394 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.394 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.394 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.394 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:35.395 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.314271+0000", 2024-04-19T02:03:35.395 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 35.066325882000001, 2024-04-19T02:03:35.395 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020466839000000001, 2024-04-19T02:03:35.395 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:35.395 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:35.395 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:35.395 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:35.395 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:35.395 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:35.395 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 117 2024-04-19T02:03:35.395 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.395 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:35.395 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.395 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:35.396 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314271+0000", 2024-04-19T02:03:35.396 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.396 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.396 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.396 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:35.396 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314271+0000", 2024-04-19T02:03:35.396 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.396 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.396 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.396 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:35.396 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314272+0000", 2024-04-19T02:03:35.396 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.514e-06 2024-04-19T02:03:35.396 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.396 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.396 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:35.397 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314284+0000", 2024-04-19T02:03:35.397 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1836e-05 2024-04-19T02:03:35.397 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.397 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.397 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:35.397 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314303+0000", 2024-04-19T02:03:35.397 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8536e-05 2024-04-19T02:03:35.397 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.397 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.397 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:35.397 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.314317+0000", 2024-04-19T02:03:35.397 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4209999999999999e-05 2024-04-19T02:03:35.397 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.397 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.397 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:35.397 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.328529+0000", 2024-04-19T02:03:35.398 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014211926999999999 2024-04-19T02:03:35.398 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.398 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.398 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:35.398 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.328658+0000", 2024-04-19T02:03:35.398 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000128763 2024-04-19T02:03:35.398 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.398 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.398 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:35.398 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.328864+0000", 2024-04-19T02:03:35.398 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020656099999999999 2024-04-19T02:03:35.398 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.398 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.398 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:35.399 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.331309+0000", 2024-04-19T02:03:35.399 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0024444879999999999 2024-04-19T02:03:35.399 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.399 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.399 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:35.399 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.334581+0000", 2024-04-19T02:03:35.399 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0032721450000000002 2024-04-19T02:03:35.399 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.399 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.399 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:35.399 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.334672+0000", 2024-04-19T02:03:35.399 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1267000000000006e-05 2024-04-19T02:03:35.399 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.399 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.399 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:35.400 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.334738+0000", 2024-04-19T02:03:35.400 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5592000000000005e-05 2024-04-19T02:03:35.400 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.400 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:35.400 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.400 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.400 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.400 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:35.400 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.791379+0000", 2024-04-19T02:03:35.400 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 34.589218242000001, 2024-04-19T02:03:35.400 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.01942953, 2024-04-19T02:03:35.400 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:35.400 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:35.401 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:35.401 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:35.401 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:35.401 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:35.401 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 136 2024-04-19T02:03:35.401 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.401 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:35.401 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.401 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:35.401 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791379+0000", 2024-04-19T02:03:35.402 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.402 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.402 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.402 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:35.402 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791379+0000", 2024-04-19T02:03:35.402 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.402 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.402 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.402 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:35.402 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791379+0000", 2024-04-19T02:03:35.402 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4000000000000001e-07 2024-04-19T02:03:35.402 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.402 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.403 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:35.403 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791387+0000", 2024-04-19T02:03:35.403 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9160000000000001e-06 2024-04-19T02:03:35.403 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.403 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.403 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:35.403 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791396+0000", 2024-04-19T02:03:35.403 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2939999999999994e-06 2024-04-19T02:03:35.403 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.403 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.403 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:35.403 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791404+0000", 2024-04-19T02:03:35.403 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6329999999999996e-06 2024-04-19T02:03:35.403 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.404 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.404 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:35.404 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.802771+0000", 2024-04-19T02:03:35.404 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011366619 2024-04-19T02:03:35.404 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.404 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.404 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:35.404 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.802918+0000", 2024-04-19T02:03:35.404 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014683900000000001 2024-04-19T02:03:35.404 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.404 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.404 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:35.404 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.803129+0000", 2024-04-19T02:03:35.404 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021161899999999999 2024-04-19T02:03:35.405 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.405 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.405 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:35.405 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.806732+0000", 2024-04-19T02:03:35.405 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0036026679999999998 2024-04-19T02:03:35.405 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.405 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.405 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:35.405 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.810544+0000", 2024-04-19T02:03:35.405 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0038118470000000002 2024-04-19T02:03:35.405 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.405 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.405 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:35.405 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.810676+0000", 2024-04-19T02:03:35.405 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013222900000000001 2024-04-19T02:03:35.406 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.406 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.406 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:35.406 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.810808+0000", 2024-04-19T02:03:35.406 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013212599999999999 2024-04-19T02:03:35.406 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.406 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:35.406 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.406 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.406 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.406 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:35.406 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.791415+0000", 2024-04-19T02:03:35.406 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 34.589181572000001, 2024-04-19T02:03:35.406 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019711117, 2024-04-19T02:03:35.406 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:35.407 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:35.407 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:35.407 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:35.407 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:35.407 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:35.407 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 137 2024-04-19T02:03:35.407 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.407 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:35.407 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.408 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:35.408 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791415+0000", 2024-04-19T02:03:35.408 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.408 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.408 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.408 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:35.408 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791415+0000", 2024-04-19T02:03:35.408 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.408 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.408 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.408 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:35.408 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791416+0000", 2024-04-19T02:03:35.408 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.23e-07 2024-04-19T02:03:35.408 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.409 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.409 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:35.409 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791584+0000", 2024-04-19T02:03:35.409 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016797599999999999 2024-04-19T02:03:35.409 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.409 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.409 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:35.409 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791600+0000", 2024-04-19T02:03:35.409 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5945999999999999e-05 2024-04-19T02:03:35.409 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.409 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.409 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:35.409 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791610+0000", 2024-04-19T02:03:35.409 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.01e-05 2024-04-19T02:03:35.409 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.410 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.410 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:35.410 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.803873+0000", 2024-04-19T02:03:35.410 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012262848999999999 2024-04-19T02:03:35.410 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.410 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.410 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:35.410 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.803995+0000", 2024-04-19T02:03:35.410 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012215899999999999 2024-04-19T02:03:35.410 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.410 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.410 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:35.410 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.804217+0000", 2024-04-19T02:03:35.411 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000222201 2024-04-19T02:03:35.411 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.411 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.411 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:35.411 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.808399+0000", 2024-04-19T02:03:35.411 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0041816910000000004 2024-04-19T02:03:35.411 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.411 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.411 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:35.411 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.810864+0000", 2024-04-19T02:03:35.411 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0024651650000000001 2024-04-19T02:03:35.411 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.411 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.411 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:35.411 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.811011+0000", 2024-04-19T02:03:35.412 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014738199999999999 2024-04-19T02:03:35.412 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.412 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.412 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:35.412 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.811126+0000", 2024-04-19T02:03:35.412 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000114925 2024-04-19T02:03:35.412 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.413 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:35.413 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.413 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.413 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.413 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:35.413 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.791625+0000", 2024-04-19T02:03:35.413 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 34.588971436999998, 2024-04-19T02:03:35.413 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019753863999999999, 2024-04-19T02:03:35.413 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:35.413 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:35.413 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:35.413 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:35.413 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:35.413 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:35.414 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 138 2024-04-19T02:03:35.414 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.414 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:35.414 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.414 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:35.414 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791625+0000", 2024-04-19T02:03:35.414 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.414 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.414 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.414 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:35.414 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791625+0000", 2024-04-19T02:03:35.414 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.414 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.414 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.415 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:35.415 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791626+0000", 2024-04-19T02:03:35.415 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8599999999999996e-07 2024-04-19T02:03:35.415 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.415 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.415 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:35.415 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791632+0000", 2024-04-19T02:03:35.415 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7200000000000003e-06 2024-04-19T02:03:35.415 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.415 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.415 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:35.415 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791641+0000", 2024-04-19T02:03:35.415 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4299999999999995e-06 2024-04-19T02:03:35.415 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.415 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.415 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:35.415 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791649+0000", 2024-04-19T02:03:35.416 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8129999999999996e-06 2024-04-19T02:03:35.416 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.416 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.416 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:35.416 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.805144+0000", 2024-04-19T02:03:35.416 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013494920000000001 2024-04-19T02:03:35.416 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.416 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.416 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:35.416 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.805387+0000", 2024-04-19T02:03:35.416 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000243188 2024-04-19T02:03:35.416 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.416 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.416 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:35.416 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.805776+0000", 2024-04-19T02:03:35.416 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00038853999999999999 2024-04-19T02:03:35.416 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.417 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.417 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:35.417 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.808536+0000", 2024-04-19T02:03:35.417 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0027598760000000001 2024-04-19T02:03:35.417 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.417 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.417 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:35.417 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.811151+0000", 2024-04-19T02:03:35.417 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0026147560000000002 2024-04-19T02:03:35.417 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.417 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.417 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:35.417 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.811275+0000", 2024-04-19T02:03:35.418 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012435199999999999 2024-04-19T02:03:35.418 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.418 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.418 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:35.418 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.811379+0000", 2024-04-19T02:03:35.418 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000104283 2024-04-19T02:03:35.418 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.418 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:35.418 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.418 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.418 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.418 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:35.419 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.791660+0000", 2024-04-19T02:03:35.419 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 34.588936410999999, 2024-04-19T02:03:35.419 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021026688000000002, 2024-04-19T02:03:35.419 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:35.419 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:35.419 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:35.419 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:35.419 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:35.419 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:35.419 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 139 2024-04-19T02:03:35.419 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.419 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:35.419 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.419 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:35.420 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791660+0000", 2024-04-19T02:03:35.420 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.420 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.420 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.420 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:35.420 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791660+0000", 2024-04-19T02:03:35.420 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.420 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.420 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.420 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:35.420 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791661+0000", 2024-04-19T02:03:35.420 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2699999999999999e-07 2024-04-19T02:03:35.420 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.421 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.421 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:35.421 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791666+0000", 2024-04-19T02:03:35.421 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4160000000000003e-06 2024-04-19T02:03:35.421 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.421 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.421 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:35.421 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791676+0000", 2024-04-19T02:03:35.421 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1169999999999993e-06 2024-04-19T02:03:35.421 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.422 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.422 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:35.422 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791683+0000", 2024-04-19T02:03:35.422 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8369999999999997e-06 2024-04-19T02:03:35.422 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.422 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.422 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:35.422 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.807160+0000", 2024-04-19T02:03:35.422 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.01547656 2024-04-19T02:03:35.422 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.422 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.422 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:35.423 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.807408+0000", 2024-04-19T02:03:35.423 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000248485 2024-04-19T02:03:35.423 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.423 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.423 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:35.423 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.807622+0000", 2024-04-19T02:03:35.423 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021381199999999999 2024-04-19T02:03:35.423 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.423 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.423 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:35.424 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812397+0000", 2024-04-19T02:03:35.424 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0047750229999999998 2024-04-19T02:03:35.424 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.424 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.424 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:35.424 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812434+0000", 2024-04-19T02:03:35.424 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6869000000000002e-05 2024-04-19T02:03:35.424 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.424 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.424 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:35.424 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812571+0000", 2024-04-19T02:03:35.424 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013649199999999999 2024-04-19T02:03:35.424 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.424 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.425 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:35.425 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812687+0000", 2024-04-19T02:03:35.425 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011635 2024-04-19T02:03:35.425 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.425 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:35.425 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.425 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.425 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.425 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:35.425 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.791694+0000", 2024-04-19T02:03:35.425 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 34.588902490999999, 2024-04-19T02:03:35.425 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021303507999999999, 2024-04-19T02:03:35.425 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:35.425 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:35.426 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:35.426 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:35.426 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:35.426 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:35.426 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 140 2024-04-19T02:03:35.426 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.426 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:35.426 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.426 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:35.426 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791694+0000", 2024-04-19T02:03:35.426 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.426 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.426 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.427 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:35.427 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791694+0000", 2024-04-19T02:03:35.427 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.427 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.427 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.427 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:35.427 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791695+0000", 2024-04-19T02:03:35.427 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2699999999999999e-07 2024-04-19T02:03:35.427 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.427 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.427 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:35.427 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791829+0000", 2024-04-19T02:03:35.428 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013436500000000001 2024-04-19T02:03:35.428 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.428 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.428 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:35.428 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791841+0000", 2024-04-19T02:03:35.428 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1707e-05 2024-04-19T02:03:35.428 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.428 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.428 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:35.428 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791850+0000", 2024-04-19T02:03:35.429 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5799999999999992e-06 2024-04-19T02:03:35.429 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.429 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.429 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:35.429 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.808635+0000", 2024-04-19T02:03:35.429 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016785133000000001 2024-04-19T02:03:35.429 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.429 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.429 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:35.429 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.808862+0000", 2024-04-19T02:03:35.429 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022693799999999999 2024-04-19T02:03:35.429 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.429 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.429 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:35.430 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.809242+0000", 2024-04-19T02:03:35.430 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00038053400000000003 2024-04-19T02:03:35.430 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.430 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.430 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:35.430 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812709+0000", 2024-04-19T02:03:35.430 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0034666929999999999 2024-04-19T02:03:35.430 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.430 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.430 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:35.430 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812796+0000", 2024-04-19T02:03:35.430 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6877000000000002e-05 2024-04-19T02:03:35.430 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.431 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.431 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:35.431 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812910+0000", 2024-04-19T02:03:35.431 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000113929 2024-04-19T02:03:35.431 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.431 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.431 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:35.431 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812998+0000", 2024-04-19T02:03:35.431 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8024999999999998e-05 2024-04-19T02:03:35.432 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.432 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:35.432 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.432 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.432 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.432 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:35.432 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.791863+0000", 2024-04-19T02:03:35.433 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 34.588733681999997, 2024-04-19T02:03:35.433 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021620092, 2024-04-19T02:03:35.433 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:35.433 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:35.433 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:35.433 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:35.434 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:35.434 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:35.434 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 141 2024-04-19T02:03:35.434 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.434 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:35.434 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.435 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:35.435 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791863+0000", 2024-04-19T02:03:35.435 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.435 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.435 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.435 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:35.436 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791863+0000", 2024-04-19T02:03:35.436 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.436 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.436 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.436 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:35.436 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791864+0000", 2024-04-19T02:03:35.436 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0599999999999999e-07 2024-04-19T02:03:35.437 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.437 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.437 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:35.437 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791871+0000", 2024-04-19T02:03:35.437 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1040000000000001e-06 2024-04-19T02:03:35.437 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.437 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.438 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:35.438 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791881+0000", 2024-04-19T02:03:35.438 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4329999999999993e-06 2024-04-19T02:03:35.438 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.438 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.439 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:35.439 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791888+0000", 2024-04-19T02:03:35.439 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7360000000000001e-06 2024-04-19T02:03:35.439 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.439 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.439 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:35.439 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.811476+0000", 2024-04-19T02:03:35.440 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019588186 2024-04-19T02:03:35.440 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.440 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.440 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:35.440 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.811595+0000", 2024-04-19T02:03:35.440 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011901200000000001 2024-04-19T02:03:35.441 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.441 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.441 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:35.441 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.811793+0000", 2024-04-19T02:03:35.441 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000197958 2024-04-19T02:03:35.441 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.441 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.441 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:35.442 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.813066+0000", 2024-04-19T02:03:35.442 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001272884 2024-04-19T02:03:35.442 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.442 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.442 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:35.442 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.813356+0000", 2024-04-19T02:03:35.443 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028986399999999997 2024-04-19T02:03:35.443 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.443 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.443 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:35.443 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.813430+0000", 2024-04-19T02:03:35.443 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3716999999999999e-05 2024-04-19T02:03:35.444 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.444 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.444 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:35.444 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.813483+0000", 2024-04-19T02:03:35.444 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3291999999999997e-05 2024-04-19T02:03:35.444 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.444 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:35.445 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.445 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.445 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.445 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:363 3.ds0 3:bdf44503:::smithi05930959-38:head [append 0~638976] snapc 9=[9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:35.445 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.746432+0000", 2024-04-19T02:03:35.445 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 8.6341652270000004, 2024-04-19T02:03:35.446 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.037983646000000003, 2024-04-19T02:03:35.446 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:35.446 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:35.446 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:35.446 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:35.447 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:35.447 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:35.447 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 363 2024-04-19T02:03:35.447 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.447 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:35.447 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.448 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:35.448 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.746432+0000", 2024-04-19T02:03:35.448 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.448 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.448 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.448 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:35.449 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.746432+0000", 2024-04-19T02:03:35.449 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.449 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.449 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.449 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:35.449 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.746435+0000", 2024-04-19T02:03:35.450 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3160000000000001e-06 2024-04-19T02:03:35.450 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.450 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.450 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:35.450 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747630+0000", 2024-04-19T02:03:35.450 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001195371 2024-04-19T02:03:35.451 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.451 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.451 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:35.451 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747664+0000", 2024-04-19T02:03:35.451 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4190000000000003e-05 2024-04-19T02:03:35.451 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.452 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.452 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:35.452 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747687+0000", 2024-04-19T02:03:35.452 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2368e-05 2024-04-19T02:03:35.452 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.453 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.453 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:35.453 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747968+0000", 2024-04-19T02:03:35.453 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028139799999999998 2024-04-19T02:03:35.453 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.454 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.454 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:35.454 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748247+0000", 2024-04-19T02:03:35.454 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00027916600000000002 2024-04-19T02:03:35.454 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.455 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.455 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:35.455 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753590+0000", 2024-04-19T02:03:35.455 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0053429920000000004 2024-04-19T02:03:35.455 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.455 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.456 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:35.456 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.782450+0000", 2024-04-19T02:03:35.456 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.028859510000000001 2024-04-19T02:03:35.456 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.456 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.456 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:35.457 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.784240+0000", 2024-04-19T02:03:35.457 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001790224 2024-04-19T02:03:35.457 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.457 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.457 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:35.457 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.784415+0000", 2024-04-19T02:03:35.458 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017511100000000001 2024-04-19T02:03:35.458 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.458 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:35.458 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.458 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.459 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.459 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.bs2 42/26 ECSubWrite(tid=133, reqid=osd.3.0:133, at_version=42'21, trim_to=0'0, roll_forward_to=39'19))", 2024-04-19T02:03:35.459 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.749670+0000", 2024-04-19T02:03:35.459 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 8.6309264389999996, 2024-04-19T02:03:35.459 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018760399000000001, 2024-04-19T02:03:35.459 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:35.460 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:35.460 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:35.460 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:35.460 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.460 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:35.461 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749670+0000", 2024-04-19T02:03:35.461 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.461 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.461 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.461 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:35.461 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749670+0000", 2024-04-19T02:03:35.462 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.462 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.462 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.462 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:35.462 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749672+0000", 2024-04-19T02:03:35.462 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3370000000000001e-06 2024-04-19T02:03:35.463 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.463 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.463 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:35.463 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749736+0000", 2024-04-19T02:03:35.463 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4350999999999994e-05 2024-04-19T02:03:35.463 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.463 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.463 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:35.464 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749763+0000", 2024-04-19T02:03:35.464 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6954999999999999e-05 2024-04-19T02:03:35.464 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.464 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.464 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:35.464 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749785+0000", 2024-04-19T02:03:35.464 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1730999999999999e-05 2024-04-19T02:03:35.464 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.464 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.465 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:35.465 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749959+0000", 2024-04-19T02:03:35.465 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000174074 2024-04-19T02:03:35.465 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.465 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.465 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:35.465 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750016+0000", 2024-04-19T02:03:35.465 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7312000000000003e-05 2024-04-19T02:03:35.465 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.466 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.466 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:35.466 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.765486+0000", 2024-04-19T02:03:35.466 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015469701000000001 2024-04-19T02:03:35.466 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.466 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.466 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:35.466 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.768431+0000", 2024-04-19T02:03:35.467 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0029449379999999998 2024-04-19T02:03:35.467 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.467 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:35.467 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.467 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.467 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.467 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.bs2 42/26 ECSubWrite(tid=135, reqid=osd.3.0:135, at_version=42'23, trim_to=0'0, roll_forward_to=39'19))", 2024-04-19T02:03:35.467 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.749932+0000", 2024-04-19T02:03:35.468 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 8.6306646590000007, 2024-04-19T02:03:35.468 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020511359, 2024-04-19T02:03:35.468 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:35.468 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:35.468 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:35.468 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:35.468 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.468 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:35.468 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749932+0000", 2024-04-19T02:03:35.468 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.468 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.468 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.468 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:35.468 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749932+0000", 2024-04-19T02:03:35.468 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.469 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.469 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.469 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:35.469 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749933+0000", 2024-04-19T02:03:35.469 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.13e-06 2024-04-19T02:03:35.469 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.469 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.469 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:35.469 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749999+0000", 2024-04-19T02:03:35.469 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5735000000000007e-05 2024-04-19T02:03:35.469 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.469 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.469 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:35.470 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750031+0000", 2024-04-19T02:03:35.470 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2419999999999998e-05 2024-04-19T02:03:35.470 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.470 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.470 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:35.470 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750051+0000", 2024-04-19T02:03:35.470 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9145999999999999e-05 2024-04-19T02:03:35.470 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.470 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.470 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:35.470 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.752530+0000", 2024-04-19T02:03:35.470 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00247964 2024-04-19T02:03:35.470 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.471 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.471 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:35.471 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.752557+0000", 2024-04-19T02:03:35.471 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6378000000000001e-05 2024-04-19T02:03:35.471 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.471 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.471 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:35.471 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.770361+0000", 2024-04-19T02:03:35.471 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017804509999999999 2024-04-19T02:03:35.471 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.471 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.471 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:35.471 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.770443+0000", 2024-04-19T02:03:35.471 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2399999999999997e-05 2024-04-19T02:03:35.471 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.472 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:35.472 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.472 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.472 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.472 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.as1 42/26 ECSubWrite(tid=82, reqid=osd.11.0:82, at_version=42'34, trim_to=0'0, roll_forward_to=39'32))", 2024-04-19T02:03:35.472 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.750430+0000", 2024-04-19T02:03:35.472 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 8.6301668009999997, 2024-04-19T02:03:35.472 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018136312000000002, 2024-04-19T02:03:35.472 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:35.472 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:35.472 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:35.472 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:35.472 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.472 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:35.472 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750430+0000", 2024-04-19T02:03:35.473 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.473 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.473 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.473 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:35.473 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750430+0000", 2024-04-19T02:03:35.473 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.473 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.473 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.473 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:35.473 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750432+0000", 2024-04-19T02:03:35.473 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8360000000000001e-06 2024-04-19T02:03:35.473 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.473 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.474 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:35.474 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750486+0000", 2024-04-19T02:03:35.474 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4413000000000002e-05 2024-04-19T02:03:35.474 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.474 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.474 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:35.474 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750519+0000", 2024-04-19T02:03:35.474 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3244000000000002e-05 2024-04-19T02:03:35.474 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.474 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.474 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:35.474 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750543+0000", 2024-04-19T02:03:35.474 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3170999999999999e-05 2024-04-19T02:03:35.474 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.475 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.475 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:35.475 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750736+0000", 2024-04-19T02:03:35.475 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000193535 2024-04-19T02:03:35.475 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.475 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.475 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:35.475 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750762+0000", 2024-04-19T02:03:35.475 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6338000000000001e-05 2024-04-19T02:03:35.475 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.475 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.475 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:35.475 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.768466+0000", 2024-04-19T02:03:35.475 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017703941000000001 2024-04-19T02:03:35.475 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.475 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.476 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:35.476 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.768566+0000", 2024-04-19T02:03:35.476 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.9834000000000007e-05 2024-04-19T02:03:35.476 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.476 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:35.476 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.476 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.476 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.476 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.5s4 42/26 ECSubWrite(tid=51, reqid=osd.10.0:51, at_version=42'27, trim_to=0'0, roll_forward_to=39'25))", 2024-04-19T02:03:35.476 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.754979+0000", 2024-04-19T02:03:35.476 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 8.6256175739999996, 2024-04-19T02:03:35.476 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.027548537000000001, 2024-04-19T02:03:35.476 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:35.476 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:35.476 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:35.477 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:35.477 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.477 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:35.477 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.754979+0000", 2024-04-19T02:03:35.477 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.477 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.477 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.477 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:35.477 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.754979+0000", 2024-04-19T02:03:35.477 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.477 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.477 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.477 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:35.477 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.754980+0000", 2024-04-19T02:03:35.477 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9999999999999996e-07 2024-04-19T02:03:35.477 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.477 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.478 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:35.478 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755050+0000", 2024-04-19T02:03:35.478 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9447000000000001e-05 2024-04-19T02:03:35.478 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.478 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.478 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:35.478 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755077+0000", 2024-04-19T02:03:35.478 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7948000000000001e-05 2024-04-19T02:03:35.478 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.478 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.478 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:35.478 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755100+0000", 2024-04-19T02:03:35.478 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2714999999999999e-05 2024-04-19T02:03:35.478 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.479 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.479 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:35.479 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760119+0000", 2024-04-19T02:03:35.479 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0050189800000000001 2024-04-19T02:03:35.479 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.479 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.479 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:35.479 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760142+0000", 2024-04-19T02:03:35.479 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2603999999999999e-05 2024-04-19T02:03:35.479 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.479 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.479 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:35.479 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.782453+0000", 2024-04-19T02:03:35.479 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022311218000000001 2024-04-19T02:03:35.480 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.480 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.480 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:35.480 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.782528+0000", 2024-04-19T02:03:35.480 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4725e-05 2024-04-19T02:03:35.480 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.480 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:35.480 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.480 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.480 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.480 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:364 3.ds0 3:bdf44503:::smithi05930959-38:head [append 0~622592] snapc 9=[9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:35.480 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.756160+0000", 2024-04-19T02:03:35.480 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 8.6244372330000001, 2024-04-19T02:03:35.480 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.034510458000000001, 2024-04-19T02:03:35.480 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:35.480 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:35.480 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:35.481 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:35.481 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:35.481 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:35.481 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 364 2024-04-19T02:03:35.481 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.481 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:35.481 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.481 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:35.481 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.756160+0000", 2024-04-19T02:03:35.481 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.481 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.481 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.481 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:35.481 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.756160+0000", 2024-04-19T02:03:35.481 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.481 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.481 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.482 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:35.482 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.756163+0000", 2024-04-19T02:03:35.482 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5130000000000001e-06 2024-04-19T02:03:35.482 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.482 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.482 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:35.482 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.757299+0000", 2024-04-19T02:03:35.482 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0011355289999999999 2024-04-19T02:03:35.482 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.482 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.482 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:35.482 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.757340+0000", 2024-04-19T02:03:35.483 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1680000000000001e-05 2024-04-19T02:03:35.483 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.483 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.483 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:35.483 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.757364+0000", 2024-04-19T02:03:35.483 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4215000000000001e-05 2024-04-19T02:03:35.483 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.483 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.483 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:35.483 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.763302+0000", 2024-04-19T02:03:35.483 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0059373480000000003 2024-04-19T02:03:35.483 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.483 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.484 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:35.484 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.768600+0000", 2024-04-19T02:03:35.484 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.005298568 2024-04-19T02:03:35.484 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.484 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.484 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:35.484 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.772526+0000", 2024-04-19T02:03:35.484 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039256450000000002 2024-04-19T02:03:35.484 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.484 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.484 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:35.484 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.790307+0000", 2024-04-19T02:03:35.484 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017781165000000002 2024-04-19T02:03:35.484 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.484 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.484 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:35.484 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.790491+0000", 2024-04-19T02:03:35.485 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000183845 2024-04-19T02:03:35.485 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.485 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.485 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:35.485 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.790670+0000", 2024-04-19T02:03:35.485 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017895000000000001 2024-04-19T02:03:35.485 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.485 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:35.485 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.485 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.485 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.485 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:365 3.ds0 3:bdf44503:::smithi05930959-38:head [append 0~147456] snapc 9=[9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:35.485 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.758315+0000", 2024-04-19T02:03:35.485 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 8.6222820569999996, 2024-04-19T02:03:35.485 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.040637806999999998, 2024-04-19T02:03:35.485 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:35.486 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:35.486 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:35.486 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:35.486 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:35.486 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:35.486 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 365 2024-04-19T02:03:35.486 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.486 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:35.486 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.486 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:35.486 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758315+0000", 2024-04-19T02:03:35.486 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.486 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.487 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.487 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:35.487 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758315+0000", 2024-04-19T02:03:35.487 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.487 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.487 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.487 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:35.487 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758317+0000", 2024-04-19T02:03:35.487 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5129999999999999e-06 2024-04-19T02:03:35.487 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.487 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.487 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:35.487 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758592+0000", 2024-04-19T02:03:35.488 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000275285 2024-04-19T02:03:35.488 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.488 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.488 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:35.488 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758625+0000", 2024-04-19T02:03:35.488 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2668e-05 2024-04-19T02:03:35.488 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.488 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.488 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:35.488 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758662+0000", 2024-04-19T02:03:35.488 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7042999999999998e-05 2024-04-19T02:03:35.488 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.488 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.488 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:35.488 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.784642+0000", 2024-04-19T02:03:35.488 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.025979510000000001 2024-04-19T02:03:35.489 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.489 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.489 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:35.489 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.785163+0000", 2024-04-19T02:03:35.489 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00052105999999999995 2024-04-19T02:03:35.489 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.489 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.489 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:35.489 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.788827+0000", 2024-04-19T02:03:35.489 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0036638109999999999 2024-04-19T02:03:35.489 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.489 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.489 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:35.489 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.792982+0000", 2024-04-19T02:03:35.490 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0041554829999999997 2024-04-19T02:03:35.490 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.490 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.490 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:35.490 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.798798+0000", 2024-04-19T02:03:35.490 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0058162379999999996 2024-04-19T02:03:35.490 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.490 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.490 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:35.490 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.798952+0000", 2024-04-19T02:03:35.490 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000154196 2024-04-19T02:03:35.490 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.490 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:35.490 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.491 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.491 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.491 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:366 3.ds0 3:bdf44503:::smithi05930959-38:head [setxattr _header (56) in=7b] snapc 9=[9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:35.491 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.758698+0000", 2024-04-19T02:03:35.491 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 8.6218991299999992, 2024-04-19T02:03:35.491 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.041876360000000001, 2024-04-19T02:03:35.491 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:35.491 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:35.491 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:35.491 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:35.491 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:35.491 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:35.491 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 366 2024-04-19T02:03:35.491 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.491 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:35.491 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.492 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:35.492 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758698+0000", 2024-04-19T02:03:35.492 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.492 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.492 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.492 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:35.492 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758698+0000", 2024-04-19T02:03:35.492 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.492 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.492 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.492 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:35.492 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758699+0000", 2024-04-19T02:03:35.492 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8160000000000001e-06 2024-04-19T02:03:35.492 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.492 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.493 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:35.493 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758711+0000", 2024-04-19T02:03:35.493 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1070000000000001e-05 2024-04-19T02:03:35.493 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.493 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.493 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:35.493 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758731+0000", 2024-04-19T02:03:35.493 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0650999999999999e-05 2024-04-19T02:03:35.493 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.493 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.493 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:35.493 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758747+0000", 2024-04-19T02:03:35.493 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5452e-05 2024-04-19T02:03:35.493 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.494 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.494 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:35.494 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.790875+0000", 2024-04-19T02:03:35.494 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.032128139999999999 2024-04-19T02:03:35.494 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.494 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.494 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:35.494 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.791060+0000", 2024-04-19T02:03:35.494 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00018499200000000001 2024-04-19T02:03:35.494 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.494 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.494 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:35.495 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.791992+0000", 2024-04-19T02:03:35.495 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00093232600000000003 2024-04-19T02:03:35.495 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.495 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.495 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:35.495 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.796307+0000", 2024-04-19T02:03:35.495 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0043147899999999998 2024-04-19T02:03:35.495 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.495 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.495 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:35.495 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.800345+0000", 2024-04-19T02:03:35.495 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.004037789 2024-04-19T02:03:35.495 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.496 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.496 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:35.496 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.800574+0000", 2024-04-19T02:03:35.496 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022933400000000001 2024-04-19T02:03:35.496 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.496 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:35.496 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.496 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.496 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.496 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:367 3.ds0 3:bdf44503:::smithi05930959-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:35.496 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.758769+0000", 2024-04-19T02:03:35.496 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 8.6218280659999991, 2024-04-19T02:03:35.496 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.049021666999999998, 2024-04-19T02:03:35.497 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:35.497 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:35.497 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:35.497 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:35.497 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:35.497 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:35.497 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 367 2024-04-19T02:03:35.497 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.497 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:35.497 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.497 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:35.497 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758769+0000", 2024-04-19T02:03:35.498 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.498 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.498 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.498 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:35.498 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758769+0000", 2024-04-19T02:03:35.498 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.498 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.498 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.498 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:35.498 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758770+0000", 2024-04-19T02:03:35.499 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5829999999999999e-06 2024-04-19T02:03:35.499 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.499 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.499 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:35.499 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758794+0000", 2024-04-19T02:03:35.500 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3408e-05 2024-04-19T02:03:35.500 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.500 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.500 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:35.500 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758815+0000", 2024-04-19T02:03:35.500 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1049000000000001e-05 2024-04-19T02:03:35.501 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.501 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.501 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:35.501 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758830+0000", 2024-04-19T02:03:35.501 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4975e-05 2024-04-19T02:03:35.501 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.502 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.502 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:35.502 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.793116+0000", 2024-04-19T02:03:35.502 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.034285806000000002 2024-04-19T02:03:35.502 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.502 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.503 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:35.503 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.793286+0000", 2024-04-19T02:03:35.503 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017076099999999999 2024-04-19T02:03:35.503 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.503 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.503 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:35.504 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.801267+0000", 2024-04-19T02:03:35.504 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0079808080000000007 2024-04-19T02:03:35.504 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.504 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.504 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:35.504 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.801431+0000", 2024-04-19T02:03:35.505 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016360499999999999 2024-04-19T02:03:35.505 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.505 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.505 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:35.505 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.807790+0000", 2024-04-19T02:03:35.505 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0063596720000000002 2024-04-19T02:03:35.506 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.506 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:35.506 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.506 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.506 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.506 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:428 3.ds0 3:b3d82b95:::smithi05930959-4:head [rollback a] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:35.510 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:30.194652+0000", 2024-04-19T02:03:35.510 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 5.1859447000000003, 2024-04-19T02:03:35.510 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020424188999999999, 2024-04-19T02:03:35.510 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:35.510 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:35.510 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:35.510 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:35.510 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:35.511 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:35.511 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 428 2024-04-19T02:03:35.511 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.511 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:35.511 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.511 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:35.511 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194652+0000", 2024-04-19T02:03:35.512 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.512 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.512 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.512 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:35.512 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194652+0000", 2024-04-19T02:03:35.512 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.512 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.512 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.513 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:35.513 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194653+0000", 2024-04-19T02:03:35.513 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8999999999999995e-07 2024-04-19T02:03:35.513 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.513 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.513 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:35.513 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194658+0000", 2024-04-19T02:03:35.513 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8629999999999999e-06 2024-04-19T02:03:35.513 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.513 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.513 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:35.513 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194670+0000", 2024-04-19T02:03:35.513 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1722999999999999e-05 2024-04-19T02:03:35.513 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.513 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.514 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:35.514 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194678+0000", 2024-04-19T02:03:35.514 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6260000000000001e-06 2024-04-19T02:03:35.514 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.514 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.514 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:35.514 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.195755+0000", 2024-04-19T02:03:35.514 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001076762 2024-04-19T02:03:35.514 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.514 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.514 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:35.514 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.195842+0000", 2024-04-19T02:03:35.514 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7409000000000003e-05 2024-04-19T02:03:35.515 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.515 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.515 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:35.515 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.196423+0000", 2024-04-19T02:03:35.515 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00058054199999999997 2024-04-19T02:03:35.515 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.515 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.515 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:35.515 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.202028+0000", 2024-04-19T02:03:35.515 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0056052280000000003 2024-04-19T02:03:35.515 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.515 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.515 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:35.515 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.214864+0000", 2024-04-19T02:03:35.515 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012835498000000001 2024-04-19T02:03:35.516 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.516 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.516 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:35.516 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.215076+0000", 2024-04-19T02:03:35.516 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021264799999999999 2024-04-19T02:03:35.516 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.516 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:35.516 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.516 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.516 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.516 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:429 3.ds0 3:b3d82b95:::smithi05930959-4:head [append 0~0] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:35.516 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:30.194690+0000", 2024-04-19T02:03:35.516 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 5.1859063650000001, 2024-04-19T02:03:35.516 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020602746000000002, 2024-04-19T02:03:35.517 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:35.517 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:35.517 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:35.517 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:35.517 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:35.517 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:35.517 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 429 2024-04-19T02:03:35.517 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.517 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:35.518 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.518 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:35.518 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194690+0000", 2024-04-19T02:03:35.518 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.518 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.518 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.518 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:35.518 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194690+0000", 2024-04-19T02:03:35.518 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:35.518 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.518 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.518 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:35.518 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194691+0000", 2024-04-19T02:03:35.519 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.23e-07 2024-04-19T02:03:35.519 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.519 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.519 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:35.519 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194695+0000", 2024-04-19T02:03:35.519 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3830000000000001e-06 2024-04-19T02:03:35.519 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.519 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.519 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:35.519 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194728+0000", 2024-04-19T02:03:35.519 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3170999999999998e-05 2024-04-19T02:03:35.519 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.519 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.519 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:35.520 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194741+0000", 2024-04-19T02:03:35.520 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3249e-05 2024-04-19T02:03:35.520 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.520 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.520 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:35.520 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.196741+0000", 2024-04-19T02:03:35.520 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0020001089999999999 2024-04-19T02:03:35.520 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.520 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.520 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:35.520 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.196828+0000", 2024-04-19T02:03:35.520 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6650999999999994e-05 2024-04-19T02:03:35.520 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.520 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.520 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:35.521 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.197349+0000", 2024-04-19T02:03:35.521 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00052079000000000003 2024-04-19T02:03:35.521 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.521 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.521 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:35.521 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.202030+0000", 2024-04-19T02:03:35.521 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0046818379999999998 2024-04-19T02:03:35.521 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.521 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.521 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:35.521 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.215221+0000", 2024-04-19T02:03:35.521 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013190485 2024-04-19T02:03:35.521 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:35.521 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:35.521 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:35.522 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.215293+0000", 2024-04-19T02:03:35.522 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2247000000000001e-05 2024-04-19T02:03:35.522 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.522 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:35.522 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.522 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:35.522 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:35.522 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:35.524 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-19T02:03:35.524 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:35.522+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:35.625 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:35.626+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:35.695 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:35.695 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:03:35.695 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-04-19T02:03:35.695 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:35.709 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-19T02:03:35.725 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:35.722+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:35.826 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:35.826+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:35.833 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:35.833 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:03:35.833 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-04-19T02:03:35.833 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-04-19T02:03:35.833 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:35.847 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-19T02:03:35.933 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:35.934+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:36.018 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:36.018 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-04-19T02:03:36.018 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-04-19T02:03:36.018 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-04-19T02:03:36.018 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.018 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s7 32/26 ECSubWrite(tid=72, reqid=client.4509.0:236, at_version=32'29, trim_to=0'0, roll_forward_to=31'28))", 2024-04-19T02:03:36.018 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:16.655346+0000", 2024-04-19T02:03:36.018 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 19.363225269000001, 2024-04-19T02:03:36.018 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011339666, 2024-04-19T02:03:36.018 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.018 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.019 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.019 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.019 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.019 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.019 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.655346+0000", 2024-04-19T02:03:36.019 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.019 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.019 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.019 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.019 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.655346+0000", 2024-04-19T02:03:36.019 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.019 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.019 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.019 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.019 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.655346+0000", 2024-04-19T02:03:36.020 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4999999999999998e-07 2024-04-19T02:03:36.020 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.020 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.020 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.020 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.655549+0000", 2024-04-19T02:03:36.020 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020310900000000001 2024-04-19T02:03:36.020 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.020 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.020 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.020 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.655578+0000", 2024-04-19T02:03:36.020 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8595999999999999e-05 2024-04-19T02:03:36.020 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.020 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.020 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.020 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.655602+0000", 2024-04-19T02:03:36.020 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3896999999999998e-05 2024-04-19T02:03:36.021 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.021 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.021 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.021 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.655777+0000", 2024-04-19T02:03:36.021 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000174935 2024-04-19T02:03:36.021 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.021 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.021 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.021 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.655803+0000", 2024-04-19T02:03:36.021 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6313e-05 2024-04-19T02:03:36.021 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.021 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.021 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.021 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.666585+0000", 2024-04-19T02:03:36.021 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010781677 2024-04-19T02:03:36.021 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.021 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.022 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.022 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:16.666685+0000", 2024-04-19T02:03:36.022 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000100789 2024-04-19T02:03:36.022 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.022 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.022 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.022 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.022 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.022 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:359 3.cs0 3:3da93e5c:::smithi05930959-42:head [copy-from ver 6] snapc 8=[8,6,5] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:36.022 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:24.774030+0000", 2024-04-19T02:03:36.022 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 11.244541027, 2024-04-19T02:03:36.022 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022176986999999999, 2024-04-19T02:03:36.022 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.022 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.023 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:36.023 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:36.023 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:36.023 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:36.023 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 359 2024-04-19T02:03:36.023 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.023 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.023 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.023 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.023 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774030+0000", 2024-04-19T02:03:36.023 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.023 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.023 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.023 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.023 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774030+0000", 2024-04-19T02:03:36.023 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.024 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.024 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.024 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.024 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774033+0000", 2024-04-19T02:03:36.024 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3859999999999999e-06 2024-04-19T02:03:36.024 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.024 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.024 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.024 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774051+0000", 2024-04-19T02:03:36.024 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8031999999999999e-05 2024-04-19T02:03:36.024 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.024 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.024 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.024 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774086+0000", 2024-04-19T02:03:36.024 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4573999999999998e-05 2024-04-19T02:03:36.024 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.025 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.025 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.025 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774108+0000", 2024-04-19T02:03:36.025 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1931000000000001e-05 2024-04-19T02:03:36.025 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.025 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.025 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.025 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774260+0000", 2024-04-19T02:03:36.025 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015161499999999999 2024-04-19T02:03:36.025 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.025 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.025 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:36.025 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774532+0000", 2024-04-19T02:03:36.025 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00027218700000000001 2024-04-19T02:03:36.025 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.025 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.026 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.026 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.791272+0000", 2024-04-19T02:03:36.026 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016740655 2024-04-19T02:03:36.026 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.026 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.026 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.026 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.794135+0000", 2024-04-19T02:03:36.026 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0028628170000000001 2024-04-19T02:03:36.026 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.026 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.026 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:36.026 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.796005+0000", 2024-04-19T02:03:36.026 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0018697 2024-04-19T02:03:36.026 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.026 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.027 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.027 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.796207+0000", 2024-04-19T02:03:36.027 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020209000000000001 2024-04-19T02:03:36.027 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.027 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.027 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.027 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.027 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.027 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:360 3.cs0 3:3da93e5c:::smithi05930959-42:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:36.027 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:24.774163+0000", 2024-04-19T02:03:36.027 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 11.244408054999999, 2024-04-19T02:03:36.027 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019807084999999999, 2024-04-19T02:03:36.027 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.027 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.027 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:36.027 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:36.028 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:36.028 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:36.028 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 360 2024-04-19T02:03:36.028 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.028 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.028 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.028 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.028 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774163+0000", 2024-04-19T02:03:36.028 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.028 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.028 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.028 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.028 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774163+0000", 2024-04-19T02:03:36.028 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.028 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.029 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.029 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.029 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774165+0000", 2024-04-19T02:03:36.029 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.513e-06 2024-04-19T02:03:36.029 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.029 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.029 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.029 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774173+0000", 2024-04-19T02:03:36.029 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.32e-06 2024-04-19T02:03:36.029 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.029 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.029 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.029 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774196+0000", 2024-04-19T02:03:36.029 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3493999999999999e-05 2024-04-19T02:03:36.029 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.029 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.029 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.030 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774213+0000", 2024-04-19T02:03:36.030 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6835999999999999e-05 2024-04-19T02:03:36.030 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.030 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.030 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.030 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774977+0000", 2024-04-19T02:03:36.030 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00076389999999999997 2024-04-19T02:03:36.030 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.030 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.030 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for blocked object", 2024-04-19T02:03:36.030 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.775221+0000", 2024-04-19T02:03:36.030 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00024397399999999999 2024-04-19T02:03:36.030 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.030 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.031 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.031 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.793417+0000", 2024-04-19T02:03:36.031 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018195896 2024-04-19T02:03:36.031 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.031 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.031 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:36.031 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.793748+0000", 2024-04-19T02:03:36.031 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000330716 2024-04-19T02:03:36.031 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.031 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.031 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.031 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.793970+0000", 2024-04-19T02:03:36.031 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022243600000000001 2024-04-19T02:03:36.031 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.031 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.032 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.032 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.032 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.032 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.ds6 42/26 ECSubWrite(tid=163, reqid=osd.8.0:163, at_version=42'14, trim_to=0'0, roll_forward_to=39'12))", 2024-04-19T02:03:36.032 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.743451+0000", 2024-04-19T02:03:36.032 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.2751205050000003, 2024-04-19T02:03:36.032 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022917335, 2024-04-19T02:03:36.032 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.032 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.032 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.032 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.032 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.032 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.032 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.743451+0000", 2024-04-19T02:03:36.033 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.033 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.033 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.033 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.033 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.743451+0000", 2024-04-19T02:03:36.033 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.033 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.033 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.033 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.033 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.743452+0000", 2024-04-19T02:03:36.033 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.33e-07 2024-04-19T02:03:36.033 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.033 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.033 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.033 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.743513+0000", 2024-04-19T02:03:36.034 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1404999999999999e-05 2024-04-19T02:03:36.034 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.034 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.034 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.034 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.743548+0000", 2024-04-19T02:03:36.034 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5187e-05 2024-04-19T02:03:36.034 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.034 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.034 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.034 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.743573+0000", 2024-04-19T02:03:36.034 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4618e-05 2024-04-19T02:03:36.034 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.034 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.034 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.035 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.743799+0000", 2024-04-19T02:03:36.035 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000226553 2024-04-19T02:03:36.035 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.035 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.035 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.035 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.743825+0000", 2024-04-19T02:03:36.035 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6101e-05 2024-04-19T02:03:36.035 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.035 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.035 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.035 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.766270+0000", 2024-04-19T02:03:36.035 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022444664999999999 2024-04-19T02:03:36.035 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.035 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.035 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.035 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.766368+0000", 2024-04-19T02:03:36.036 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7873000000000003e-05 2024-04-19T02:03:36.036 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.036 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.036 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.036 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.036 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.036 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s7 42/26 ECSubWrite(tid=132, reqid=osd.3.0:132, at_version=42'63, trim_to=0'0, roll_forward_to=40'61))", 2024-04-19T02:03:36.036 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.747459+0000", 2024-04-19T02:03:36.036 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.2711121809999995, 2024-04-19T02:03:36.036 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.023151675, 2024-04-19T02:03:36.036 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.036 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.036 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.037 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.037 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.037 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.037 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747459+0000", 2024-04-19T02:03:36.037 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.037 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.037 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.037 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.037 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747459+0000", 2024-04-19T02:03:36.037 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.037 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.037 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.037 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.038 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747460+0000", 2024-04-19T02:03:36.038 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7000000000000004e-07 2024-04-19T02:03:36.038 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.038 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.038 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.038 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747515+0000", 2024-04-19T02:03:36.038 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5248999999999999e-05 2024-04-19T02:03:36.038 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.038 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.038 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.038 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747546+0000", 2024-04-19T02:03:36.038 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1244000000000001e-05 2024-04-19T02:03:36.038 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.038 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.038 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.039 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747569+0000", 2024-04-19T02:03:36.039 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2835000000000001e-05 2024-04-19T02:03:36.039 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.039 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.039 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.039 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747776+0000", 2024-04-19T02:03:36.039 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020755099999999999 2024-04-19T02:03:36.039 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.039 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.039 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.039 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747802+0000", 2024-04-19T02:03:36.039 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5694000000000002e-05 2024-04-19T02:03:36.039 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.039 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.039 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.040 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.770557+0000", 2024-04-19T02:03:36.040 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022754936 2024-04-19T02:03:36.040 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.040 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.040 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.040 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.770611+0000", 2024-04-19T02:03:36.040 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3495999999999997e-05 2024-04-19T02:03:36.040 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.040 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.040 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.040 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.040 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.040 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.5s3 42/26 ECSubWrite(tid=51, reqid=osd.10.0:51, at_version=42'27, trim_to=0'0, roll_forward_to=39'25))", 2024-04-19T02:03:36.040 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.750076+0000", 2024-04-19T02:03:36.040 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.2684948309999999, 2024-04-19T02:03:36.041 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020350849000000001, 2024-04-19T02:03:36.041 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.041 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.041 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.041 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.041 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.041 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.041 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750076+0000", 2024-04-19T02:03:36.041 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.041 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.041 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.041 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.041 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750076+0000", 2024-04-19T02:03:36.041 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.041 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.041 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.042 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.042 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750077+0000", 2024-04-19T02:03:36.042 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7000000000000004e-07 2024-04-19T02:03:36.042 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.042 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.042 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.042 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750133+0000", 2024-04-19T02:03:36.042 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6119000000000003e-05 2024-04-19T02:03:36.042 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.042 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.042 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.042 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750163+0000", 2024-04-19T02:03:36.042 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0328000000000002e-05 2024-04-19T02:03:36.043 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.043 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.043 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.043 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750220+0000", 2024-04-19T02:03:36.043 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6325e-05 2024-04-19T02:03:36.043 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.043 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.043 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.043 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750376+0000", 2024-04-19T02:03:36.043 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015590100000000001 2024-04-19T02:03:36.043 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.043 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.043 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.043 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750401+0000", 2024-04-19T02:03:36.044 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5635000000000001e-05 2024-04-19T02:03:36.044 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.044 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.044 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.044 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.770323+0000", 2024-04-19T02:03:36.044 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019921464999999999 2024-04-19T02:03:36.044 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.044 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.044 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.044 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.770427+0000", 2024-04-19T02:03:36.044 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000104306 2024-04-19T02:03:36.044 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.044 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.044 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.045 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.045 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.045 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.as3 42/26 ECSubWrite(tid=82, reqid=osd.11.0:82, at_version=42'34, trim_to=0'0, roll_forward_to=39'32))", 2024-04-19T02:03:36.045 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.750214+0000", 2024-04-19T02:03:36.045 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.2683568610000009, 2024-04-19T02:03:36.045 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019187731999999999, 2024-04-19T02:03:36.045 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.045 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.045 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.045 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.045 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.045 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.045 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750214+0000", 2024-04-19T02:03:36.045 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.046 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.046 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.046 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.046 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750214+0000", 2024-04-19T02:03:36.046 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.046 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.046 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.046 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.046 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750215+0000", 2024-04-19T02:03:36.046 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4000000000000001e-07 2024-04-19T02:03:36.046 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.046 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.046 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.046 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750268+0000", 2024-04-19T02:03:36.047 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3389e-05 2024-04-19T02:03:36.047 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.047 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.047 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.047 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750299+0000", 2024-04-19T02:03:36.047 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0914000000000003e-05 2024-04-19T02:03:36.047 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.047 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.047 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.047 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750322+0000", 2024-04-19T02:03:36.047 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2575e-05 2024-04-19T02:03:36.047 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.047 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.047 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.048 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750578+0000", 2024-04-19T02:03:36.048 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00025623300000000002 2024-04-19T02:03:36.048 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.048 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.048 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.048 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750606+0000", 2024-04-19T02:03:36.048 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7568000000000001e-05 2024-04-19T02:03:36.048 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.048 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.048 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.048 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.769319+0000", 2024-04-19T02:03:36.048 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018712909 2024-04-19T02:03:36.048 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.048 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.049 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.049 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.769402+0000", 2024-04-19T02:03:36.049 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3404000000000003e-05 2024-04-19T02:03:36.049 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.049 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.049 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.049 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.049 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.049 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s7 42/26 ECSubWrite(tid=134, reqid=osd.3.0:134, at_version=42'65, trim_to=0'0, roll_forward_to=40'61))", 2024-04-19T02:03:36.049 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.750993+0000", 2024-04-19T02:03:36.049 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.2675776889999995, 2024-04-19T02:03:36.049 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021026969, 2024-04-19T02:03:36.049 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.049 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.049 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.050 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.050 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.050 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.050 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750993+0000", 2024-04-19T02:03:36.050 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.050 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.050 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.050 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.050 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750993+0000", 2024-04-19T02:03:36.050 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.050 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.050 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.050 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.050 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750995+0000", 2024-04-19T02:03:36.050 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.883e-06 2024-04-19T02:03:36.051 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.051 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.051 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.051 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751070+0000", 2024-04-19T02:03:36.051 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4981000000000006e-05 2024-04-19T02:03:36.051 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.051 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.051 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.051 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751100+0000", 2024-04-19T02:03:36.051 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9944999999999999e-05 2024-04-19T02:03:36.051 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.051 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.051 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.051 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751122+0000", 2024-04-19T02:03:36.051 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1810999999999998e-05 2024-04-19T02:03:36.051 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.051 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.052 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.052 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.762958+0000", 2024-04-19T02:03:36.052 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011836187 2024-04-19T02:03:36.052 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.052 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.052 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.052 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.762984+0000", 2024-04-19T02:03:36.052 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5420999999999999e-05 2024-04-19T02:03:36.052 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.052 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.052 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.052 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.771936+0000", 2024-04-19T02:03:36.052 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0089524319999999998 2024-04-19T02:03:36.052 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.052 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.052 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.053 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.772020+0000", 2024-04-19T02:03:36.053 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4308999999999995e-05 2024-04-19T02:03:36.053 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.053 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.053 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.053 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.053 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.053 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.8s5 42/26 ECSubWrite(tid=71, reqid=osd.4.0:71, at_version=42'33, trim_to=0'0, roll_forward_to=37'31))", 2024-04-19T02:03:36.053 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.753652+0000", 2024-04-19T02:03:36.053 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.2649188319999993, 2024-04-19T02:03:36.053 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012125554, 2024-04-19T02:03:36.053 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.053 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.053 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.053 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.053 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.053 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.054 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753652+0000", 2024-04-19T02:03:36.054 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.054 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.054 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.054 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.054 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753652+0000", 2024-04-19T02:03:36.054 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.054 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.054 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.054 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.054 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753654+0000", 2024-04-19T02:03:36.054 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.35e-06 2024-04-19T02:03:36.054 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.054 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.054 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.054 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753719+0000", 2024-04-19T02:03:36.055 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5501999999999997e-05 2024-04-19T02:03:36.055 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.055 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.055 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.055 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753747+0000", 2024-04-19T02:03:36.055 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7691e-05 2024-04-19T02:03:36.055 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.055 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.055 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.055 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753772+0000", 2024-04-19T02:03:36.055 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4825e-05 2024-04-19T02:03:36.055 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.055 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.055 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.055 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.754382+0000", 2024-04-19T02:03:36.055 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00061038900000000003 2024-04-19T02:03:36.055 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.056 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.056 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.056 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.756197+0000", 2024-04-19T02:03:36.056 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0018153589999999999 2024-04-19T02:03:36.056 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.056 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.056 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.056 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.765586+0000", 2024-04-19T02:03:36.056 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0093882529999999992 2024-04-19T02:03:36.056 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.056 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.056 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.056 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.765778+0000", 2024-04-19T02:03:36.056 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00019218499999999999 2024-04-19T02:03:36.056 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.056 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.057 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.057 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.057 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.057 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:377 3.cs0 3:3da93e5c:::smithi05930959-42:head [read 0~589824,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:36.057 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.759004+0000", 2024-04-19T02:03:36.057 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.2595671740000007, 2024-04-19T02:03:36.057 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.035358317, 2024-04-19T02:03:36.057 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.057 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.057 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:36.057 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:36.057 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:36.057 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:36.057 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 377 2024-04-19T02:03:36.057 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.057 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.058 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.058 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.058 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759004+0000", 2024-04-19T02:03:36.058 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.058 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.058 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.058 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.058 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759004+0000", 2024-04-19T02:03:36.058 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.058 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.058 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.058 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.058 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759008+0000", 2024-04-19T02:03:36.058 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9190000000000001e-06 2024-04-19T02:03:36.058 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.059 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.059 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.059 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759024+0000", 2024-04-19T02:03:36.059 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6195e-05 2024-04-19T02:03:36.059 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.059 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.059 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.059 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759058+0000", 2024-04-19T02:03:36.059 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4238e-05 2024-04-19T02:03:36.059 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.059 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.059 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.059 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759080+0000", 2024-04-19T02:03:36.059 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1237999999999999e-05 2024-04-19T02:03:36.060 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.060 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.060 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.060 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.771701+0000", 2024-04-19T02:03:36.060 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012621098000000001 2024-04-19T02:03:36.060 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.060 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.060 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:36.060 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.771949+0000", 2024-04-19T02:03:36.060 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00024814599999999998 2024-04-19T02:03:36.060 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.060 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.060 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.061 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.788299+0000", 2024-04-19T02:03:36.061 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016350138 2024-04-19T02:03:36.061 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.061 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.061 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:36.061 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.788497+0000", 2024-04-19T02:03:36.061 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00019762100000000001 2024-04-19T02:03:36.061 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.061 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.061 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.061 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.794362+0000", 2024-04-19T02:03:36.061 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0058657240000000001 2024-04-19T02:03:36.061 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.061 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.062 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.062 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.062 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.062 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:378 3.cs0 3:3da93e5c:::smithi05930959-42:head [read 0~589824,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:36.062 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.759134+0000", 2024-04-19T02:03:36.062 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.2594373710000006, 2024-04-19T02:03:36.062 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.044044933000000001, 2024-04-19T02:03:36.062 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.062 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.062 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:36.062 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:36.062 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:36.062 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:36.062 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 378 2024-04-19T02:03:36.063 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.063 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.063 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.063 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.063 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759134+0000", 2024-04-19T02:03:36.063 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.063 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.063 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.063 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.063 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759134+0000", 2024-04-19T02:03:36.063 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.063 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.063 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.063 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.064 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759136+0000", 2024-04-19T02:03:36.064 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8029999999999999e-06 2024-04-19T02:03:36.064 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.064 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.064 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.064 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759145+0000", 2024-04-19T02:03:36.064 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5789999999999993e-06 2024-04-19T02:03:36.064 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.064 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.064 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.064 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759166+0000", 2024-04-19T02:03:36.064 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0815e-05 2024-04-19T02:03:36.064 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.064 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.064 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.064 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759187+0000", 2024-04-19T02:03:36.065 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1299000000000001e-05 2024-04-19T02:03:36.065 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.065 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.065 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.065 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.772621+0000", 2024-04-19T02:03:36.065 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013434032 2024-04-19T02:03:36.065 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.065 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.065 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:36.065 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.772918+0000", 2024-04-19T02:03:36.065 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00029657400000000002 2024-04-19T02:03:36.065 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.065 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.065 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.065 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.794802+0000", 2024-04-19T02:03:36.065 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021883785999999999 2024-04-19T02:03:36.065 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.066 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.066 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:36.066 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.795248+0000", 2024-04-19T02:03:36.066 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00044615099999999999 2024-04-19T02:03:36.066 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.066 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.066 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.066 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.803179+0000", 2024-04-19T02:03:36.066 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0079308940000000008 2024-04-19T02:03:36.066 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.066 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.066 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.066 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.066 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.067 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:379 3.cs0 3:3da93e5c:::smithi05930959-42:head [read 0~589824,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:36.067 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.759209+0000", 2024-04-19T02:03:36.067 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.2593616339999993, 2024-04-19T02:03:36.067 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.045345979000000002, 2024-04-19T02:03:36.067 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.067 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.067 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:36.067 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:36.067 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:36.067 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:36.067 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 379 2024-04-19T02:03:36.067 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.067 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.067 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.068 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.068 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759209+0000", 2024-04-19T02:03:36.068 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.068 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.068 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.068 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.068 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759209+0000", 2024-04-19T02:03:36.068 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.068 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.068 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.068 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.068 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759210+0000", 2024-04-19T02:03:36.068 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9700000000000005e-07 2024-04-19T02:03:36.068 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.068 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.068 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.068 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759218+0000", 2024-04-19T02:03:36.069 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9589999999999995e-06 2024-04-19T02:03:36.069 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.069 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.069 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.069 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759236+0000", 2024-04-19T02:03:36.069 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7209000000000001e-05 2024-04-19T02:03:36.069 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.069 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.069 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.069 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759249+0000", 2024-04-19T02:03:36.069 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3956e-05 2024-04-19T02:03:36.069 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.069 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.069 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.069 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.773037+0000", 2024-04-19T02:03:36.069 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013787415000000001 2024-04-19T02:03:36.070 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.070 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.070 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:36.070 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.773251+0000", 2024-04-19T02:03:36.070 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021379099999999999 2024-04-19T02:03:36.070 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.070 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.070 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.070 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.797127+0000", 2024-04-19T02:03:36.070 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.023876128999999999 2024-04-19T02:03:36.070 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.070 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.070 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:36.070 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.797749+0000", 2024-04-19T02:03:36.070 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00062207600000000005 2024-04-19T02:03:36.071 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.071 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.071 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.071 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.804555+0000", 2024-04-19T02:03:36.071 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0068065469999999996 2024-04-19T02:03:36.071 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.071 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.071 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.071 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.071 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.071 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.ds6 42/26 ECSubWrite(tid=164, reqid=client.4509.0:363, at_version=42'15, trim_to=0'0, roll_forward_to=39'12))", 2024-04-19T02:03:36.071 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.760370+0000", 2024-04-19T02:03:36.071 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.2582009640000003, 2024-04-19T02:03:36.071 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010134341999999999, 2024-04-19T02:03:36.072 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.072 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.072 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.072 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.072 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.072 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.072 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760370+0000", 2024-04-19T02:03:36.072 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.072 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.072 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.072 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.072 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760370+0000", 2024-04-19T02:03:36.072 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.072 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.073 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.073 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.073 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760372+0000", 2024-04-19T02:03:36.073 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0229999999999999e-06 2024-04-19T02:03:36.073 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.073 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.073 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.073 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760699+0000", 2024-04-19T02:03:36.073 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00032690099999999999 2024-04-19T02:03:36.073 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.073 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.073 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.073 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760733+0000", 2024-04-19T02:03:36.073 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3927000000000002e-05 2024-04-19T02:03:36.074 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.074 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.074 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.074 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760756+0000", 2024-04-19T02:03:36.074 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3292e-05 2024-04-19T02:03:36.074 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.074 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.074 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.074 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.763173+0000", 2024-04-19T02:03:36.074 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0024166389999999999 2024-04-19T02:03:36.074 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.075 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.075 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.075 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.763199+0000", 2024-04-19T02:03:36.075 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6010999999999999e-05 2024-04-19T02:03:36.075 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.075 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.075 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.075 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.770435+0000", 2024-04-19T02:03:36.075 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0072360549999999999 2024-04-19T02:03:36.075 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.075 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.075 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.075 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.770504+0000", 2024-04-19T02:03:36.075 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9493999999999996e-05 2024-04-19T02:03:36.075 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.076 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.076 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.076 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.076 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.076 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.5s3 42/26 ECSubWrite(tid=52, reqid=osd.10.0:52, at_version=42'29, trim_to=0'0, roll_forward_to=39'25))", 2024-04-19T02:03:36.076 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.760812+0000", 2024-04-19T02:03:36.076 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.257758699, 2024-04-19T02:03:36.076 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010763234, 2024-04-19T02:03:36.076 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.076 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.076 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.076 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.076 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.076 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.076 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760812+0000", 2024-04-19T02:03:36.077 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.077 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.077 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.077 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.077 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760812+0000", 2024-04-19T02:03:36.077 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.077 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.077 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.077 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.077 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760814+0000", 2024-04-19T02:03:36.077 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7269999999999999e-06 2024-04-19T02:03:36.077 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.077 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.077 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.077 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760866+0000", 2024-04-19T02:03:36.078 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1858999999999999e-05 2024-04-19T02:03:36.078 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.078 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.078 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.078 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760897+0000", 2024-04-19T02:03:36.078 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1068000000000001e-05 2024-04-19T02:03:36.078 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.078 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.078 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.078 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760920+0000", 2024-04-19T02:03:36.078 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2588000000000001e-05 2024-04-19T02:03:36.078 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.078 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.078 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.079 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.765904+0000", 2024-04-19T02:03:36.079 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.004983889 2024-04-19T02:03:36.079 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.079 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.079 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.079 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.765931+0000", 2024-04-19T02:03:36.079 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7898e-05 2024-04-19T02:03:36.079 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.079 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.079 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.079 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.771493+0000", 2024-04-19T02:03:36.079 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0055615250000000003 2024-04-19T02:03:36.079 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.079 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.079 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.079 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.771576+0000", 2024-04-19T02:03:36.080 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2680000000000001e-05 2024-04-19T02:03:36.080 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.080 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.080 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.080 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.080 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.080 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.es7 42/26 ECSubWrite(tid=32, reqid=osd.7.0:32, at_version=42'25, trim_to=0'0, roll_forward_to=40'23))", 2024-04-19T02:03:36.080 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.761131+0000", 2024-04-19T02:03:36.080 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.2574402770000006, 2024-04-19T02:03:36.080 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010525209000000001, 2024-04-19T02:03:36.080 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.080 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.080 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.080 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.080 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.081 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.081 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761131+0000", 2024-04-19T02:03:36.081 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.081 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.081 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.081 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.081 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761131+0000", 2024-04-19T02:03:36.081 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.081 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.081 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.081 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.081 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761132+0000", 2024-04-19T02:03:36.081 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1699999999999997e-07 2024-04-19T02:03:36.081 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.081 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.082 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.082 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761187+0000", 2024-04-19T02:03:36.082 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5186000000000003e-05 2024-04-19T02:03:36.082 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.082 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.082 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.082 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761216+0000", 2024-04-19T02:03:36.082 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9213999999999998e-05 2024-04-19T02:03:36.082 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.082 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.082 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.082 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761238+0000", 2024-04-19T02:03:36.082 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2035000000000002e-05 2024-04-19T02:03:36.082 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.083 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.083 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.083 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761391+0000", 2024-04-19T02:03:36.083 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000152841 2024-04-19T02:03:36.083 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.083 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.083 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.083 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761417+0000", 2024-04-19T02:03:36.083 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6548000000000001e-05 2024-04-19T02:03:36.084 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.084 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.084 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.084 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.771598+0000", 2024-04-19T02:03:36.084 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010180829000000001 2024-04-19T02:03:36.084 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.084 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.084 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.084 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.771656+0000", 2024-04-19T02:03:36.084 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7839e-05 2024-04-19T02:03:36.084 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.084 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.084 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.084 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.084 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.085 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.ds6 42/26 ECSubWrite(tid=165, reqid=client.4509.0:364, at_version=42'16, trim_to=0'0, roll_forward_to=39'12))", 2024-04-19T02:03:36.085 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.774419+0000", 2024-04-19T02:03:36.085 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.2441518360000003, 2024-04-19T02:03:36.085 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0085658109999999996, 2024-04-19T02:03:36.085 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.085 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.085 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.085 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.085 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.085 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.085 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774419+0000", 2024-04-19T02:03:36.085 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.085 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.085 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.086 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.086 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774419+0000", 2024-04-19T02:03:36.086 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.086 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.086 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.086 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.086 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774421+0000", 2024-04-19T02:03:36.086 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3939999999999999e-06 2024-04-19T02:03:36.086 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.086 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.086 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.086 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774715+0000", 2024-04-19T02:03:36.086 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0002943 2024-04-19T02:03:36.086 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.086 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.087 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.087 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774744+0000", 2024-04-19T02:03:36.087 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9437999999999998e-05 2024-04-19T02:03:36.087 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.087 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.087 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.087 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774766+0000", 2024-04-19T02:03:36.087 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1404999999999999e-05 2024-04-19T02:03:36.087 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.087 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.087 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.087 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774907+0000", 2024-04-19T02:03:36.087 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000141655 2024-04-19T02:03:36.087 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.088 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.088 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.088 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774944+0000", 2024-04-19T02:03:36.088 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6216e-05 2024-04-19T02:03:36.088 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.088 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.088 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.088 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.782910+0000", 2024-04-19T02:03:36.088 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0079663189999999995 2024-04-19T02:03:36.088 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.088 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.088 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.088 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.782985+0000", 2024-04-19T02:03:36.088 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5084000000000001e-05 2024-04-19T02:03:36.088 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.089 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.089 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.089 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.089 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.089 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s7 46/26 ECSubWrite(tid=163, reqid=client.4509.0:425, at_version=46'78, trim_to=0'0, roll_forward_to=46'75))", 2024-04-19T02:03:36.089 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:30.198536+0000", 2024-04-19T02:03:36.089 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 5.8200352610000001, 2024-04-19T02:03:36.089 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012590654999999999, 2024-04-19T02:03:36.089 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.089 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.089 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.089 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.089 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.089 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.089 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198536+0000", 2024-04-19T02:03:36.090 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.090 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.090 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.090 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.090 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198536+0000", 2024-04-19T02:03:36.090 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.090 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.090 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.090 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.090 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198537+0000", 2024-04-19T02:03:36.090 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.9000000000000005e-07 2024-04-19T02:03:36.090 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.090 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.091 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.091 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198582+0000", 2024-04-19T02:03:36.091 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5222999999999997e-05 2024-04-19T02:03:36.091 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.091 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.091 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.091 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198606+0000", 2024-04-19T02:03:36.091 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4091999999999999e-05 2024-04-19T02:03:36.091 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.091 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.091 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.091 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198624+0000", 2024-04-19T02:03:36.091 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8189000000000002e-05 2024-04-19T02:03:36.092 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.092 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.092 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.092 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.199934+0000", 2024-04-19T02:03:36.092 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0013098490000000001 2024-04-19T02:03:36.092 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.092 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.092 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.092 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.199958+0000", 2024-04-19T02:03:36.092 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4153999999999999e-05 2024-04-19T02:03:36.092 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.092 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.092 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.092 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.211045+0000", 2024-04-19T02:03:36.092 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011086597 2024-04-19T02:03:36.093 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.093 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.093 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.093 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.211126+0000", 2024-04-19T02:03:36.093 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1561000000000001e-05 2024-04-19T02:03:36.093 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.093 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.093 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.093 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.093 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.093 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s7 46/26 ECSubWrite(tid=164, reqid=client.4509.0:426, at_version=46'79, trim_to=0'0, roll_forward_to=46'75))", 2024-04-19T02:03:36.093 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:30.199915+0000", 2024-04-19T02:03:36.093 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 5.8186556439999997, 2024-04-19T02:03:36.093 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011271487, 2024-04-19T02:03:36.093 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.093 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.094 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.094 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.094 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.094 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.094 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.199915+0000", 2024-04-19T02:03:36.094 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.094 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.094 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.094 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.094 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.199915+0000", 2024-04-19T02:03:36.094 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.094 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.094 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.094 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.094 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.199917+0000", 2024-04-19T02:03:36.095 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.623e-06 2024-04-19T02:03:36.095 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.095 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.095 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.095 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.199957+0000", 2024-04-19T02:03:36.095 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0364000000000002e-05 2024-04-19T02:03:36.095 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.095 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.095 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.095 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.199986+0000", 2024-04-19T02:03:36.095 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9025000000000001e-05 2024-04-19T02:03:36.095 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.095 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.095 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.095 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.200015+0000", 2024-04-19T02:03:36.096 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8748e-05 2024-04-19T02:03:36.096 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.096 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.096 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.096 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.200669+0000", 2024-04-19T02:03:36.096 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00065353099999999995 2024-04-19T02:03:36.096 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.096 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.096 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.096 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.200679+0000", 2024-04-19T02:03:36.096 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0413e-05 2024-04-19T02:03:36.096 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.096 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.096 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.096 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.211133+0000", 2024-04-19T02:03:36.097 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010453954 2024-04-19T02:03:36.097 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.097 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.097 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.097 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.211187+0000", 2024-04-19T02:03:36.097 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3829000000000002e-05 2024-04-19T02:03:36.097 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.097 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.097 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.097 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.097 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.097 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.ds6 46/26 ECSubWrite(tid=191, reqid=client.4509.0:429, at_version=46'38, trim_to=0'0, roll_forward_to=46'35))", 2024-04-19T02:03:36.097 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:30.201330+0000", 2024-04-19T02:03:36.098 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 5.8172406890000001, 2024-04-19T02:03:36.098 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0092012120000000003, 2024-04-19T02:03:36.098 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.098 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.098 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.098 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.098 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.098 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.098 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.201330+0000", 2024-04-19T02:03:36.098 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.098 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.098 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.098 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.098 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.201330+0000", 2024-04-19T02:03:36.099 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.099 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.099 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.099 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.099 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.201332+0000", 2024-04-19T02:03:36.099 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6160000000000001e-06 2024-04-19T02:03:36.099 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.099 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.099 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.099 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.201375+0000", 2024-04-19T02:03:36.099 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3021e-05 2024-04-19T02:03:36.099 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.099 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.099 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.100 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.201406+0000", 2024-04-19T02:03:36.100 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0617999999999997e-05 2024-04-19T02:03:36.100 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.100 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.100 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.100 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.201427+0000", 2024-04-19T02:03:36.100 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1228000000000001e-05 2024-04-19T02:03:36.100 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.100 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.100 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.100 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.201676+0000", 2024-04-19T02:03:36.100 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00024866999999999999 2024-04-19T02:03:36.100 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.100 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.100 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.101 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.201700+0000", 2024-04-19T02:03:36.101 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4564999999999999e-05 2024-04-19T02:03:36.101 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.101 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.101 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.101 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.210457+0000", 2024-04-19T02:03:36.101 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0087572520000000001 2024-04-19T02:03:36.101 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.101 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.101 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.101 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.210532+0000", 2024-04-19T02:03:36.101 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4241999999999998e-05 2024-04-19T02:03:36.101 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.101 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.101 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.102 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.102 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.102 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s7 47/26 ECSubWrite(tid=184, reqid=osd.3.0:182, at_version=0'0, trim_to=0'0, roll_forward_to=47'89))", 2024-04-19T02:03:36.102 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.117883+0000", 2024-04-19T02:03:36.102 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4.9006877600000003, 2024-04-19T02:03:36.102 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0084990240000000009, 2024-04-19T02:03:36.102 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.102 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.102 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.102 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.102 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.102 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.102 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.117883+0000", 2024-04-19T02:03:36.102 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.103 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.103 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.103 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.103 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.117883+0000", 2024-04-19T02:03:36.103 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.103 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.103 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.103 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.103 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.117885+0000", 2024-04-19T02:03:36.103 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9429999999999999e-06 2024-04-19T02:03:36.103 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.103 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.103 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.103 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.117911+0000", 2024-04-19T02:03:36.104 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6015000000000001e-05 2024-04-19T02:03:36.104 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.104 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.104 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.104 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.117942+0000", 2024-04-19T02:03:36.104 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0744000000000002e-05 2024-04-19T02:03:36.104 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.104 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.104 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.104 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.117964+0000", 2024-04-19T02:03:36.104 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2178999999999999e-05 2024-04-19T02:03:36.104 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.104 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.104 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.104 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.120537+0000", 2024-04-19T02:03:36.105 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0025723590000000002 2024-04-19T02:03:36.105 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.105 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.105 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.105 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.120568+0000", 2024-04-19T02:03:36.105 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1392e-05 2024-04-19T02:03:36.105 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.105 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.105 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.105 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.126308+0000", 2024-04-19T02:03:36.105 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0057397869999999997 2024-04-19T02:03:36.105 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.105 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.105 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.105 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.126382+0000", 2024-04-19T02:03:36.106 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4604999999999995e-05 2024-04-19T02:03:36.106 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.106 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.106 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.106 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.106 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.106 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:36.107 INFO:teuthology.orchestra.run.smithi059.stderr:nodeep-scrub is set 2024-04-19T02:03:36.108 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.9 to 1 (10000) 2024-04-19T02:03:36.108 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-19T02:03:36.109 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:36.106+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:36.112 INFO:tasks.rados.rados.0.smithi059.stdout:200: done (2 left) 2024-04-19T02:03:36.112 INFO:tasks.rados.rados.0.smithi059.stdout:202: done (1 left) 2024-04-19T02:03:36.112 INFO:tasks.rados.rados.0.smithi059.stdout:205: done (0 left) 2024-04-19T02:03:36.112 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 25 current snap is 13 2024-04-19T02:03:36.112 INFO:tasks.rados.rados.0.smithi059.stdout:206: seq_num 101 ranges {1294336=540672,1835008=507904,2342912=622592,2965504=32768} 2024-04-19T02:03:36.127 INFO:tasks.rados.rados.0.smithi059.stdout:206: writing smithi05930959-25 from 1294336 to 1835008 tid 1 2024-04-19T02:03:36.128 INFO:tasks.rados.rados.0.smithi059.stdout:206: writing smithi05930959-25 from 1835008 to 2342912 tid 2 2024-04-19T02:03:36.132 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-04-19T02:03:36.133 INFO:tasks.rados.rados.0.smithi059.stdout:206: writing smithi05930959-25 from 2342912 to 2965504 tid 3 2024-04-19T02:03:36.133 INFO:tasks.rados.rados.0.smithi059.stdout:206: writing smithi05930959-25 from 2965504 to 2998272 tid 4 2024-04-19T02:03:36.133 INFO:tasks.rados.rados.0.smithi059.stdout:207: copy_from oid 1 from oid 18 current snap is 13 2024-04-19T02:03:36.133 INFO:tasks.rados.rados.0.smithi059.stdout:208: snap_remove snap 9 2024-04-19T02:03:36.208 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:36.206+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:36.282 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:36.282 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:03:36.282 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-04-19T02:03:36.282 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:36.296 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-19T02:03:36.309 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:36.306+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:36.410 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:36.410+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:36.417 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:36.417 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:03:36.417 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-04-19T02:03:36.417 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-04-19T02:03:36.417 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:36.431 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-19T02:03:36.505 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T02:03:36.511 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:03:36.514+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:03:36.602 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:36.602 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-04-19T02:03:36.602 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-04-19T02:03:36.602 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-04-19T02:03:36.602 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.602 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:293 3.5s0 3:a41c791e:::smithi05930959-15:head [delete] snapc 6=[6,5] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e37)", 2024-04-19T02:03:36.602 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:21.689881+0000", 2024-04-19T02:03:36.602 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 14.912857456999999, 2024-04-19T02:03:36.602 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011022188, 2024-04-19T02:03:36.603 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.603 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.603 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:36.603 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:36.603 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:36.603 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:36.603 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 293 2024-04-19T02:03:36.603 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.603 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.603 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.603 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.603 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.689881+0000", 2024-04-19T02:03:36.603 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.603 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.603 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.604 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.604 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.689881+0000", 2024-04-19T02:03:36.604 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.604 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.604 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.604 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.604 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.689883+0000", 2024-04-19T02:03:36.604 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6170000000000001e-06 2024-04-19T02:03:36.604 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.604 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.604 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.604 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.689896+0000", 2024-04-19T02:03:36.604 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2826000000000001e-05 2024-04-19T02:03:36.605 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.605 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.605 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.605 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.689930+0000", 2024-04-19T02:03:36.605 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4289999999999999e-05 2024-04-19T02:03:36.605 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.605 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.605 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.605 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.689949+0000", 2024-04-19T02:03:36.605 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8921000000000001e-05 2024-04-19T02:03:36.605 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.605 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.605 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.605 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.690106+0000", 2024-04-19T02:03:36.605 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000156496 2024-04-19T02:03:36.606 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.606 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.606 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op must wait for map", 2024-04-19T02:03:36.606 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.690123+0000", 2024-04-19T02:03:36.606 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6949e-05 2024-04-19T02:03:36.606 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.606 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.606 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.606 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.693813+0000", 2024-04-19T02:03:36.606 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0036899419999999999 2024-04-19T02:03:36.606 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.606 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.606 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:36.606 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.693962+0000", 2024-04-19T02:03:36.607 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014913 2024-04-19T02:03:36.607 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.607 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.607 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.607 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.695407+0000", 2024-04-19T02:03:36.607 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001444823 2024-04-19T02:03:36.607 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.607 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.607 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.607 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.698101+0000", 2024-04-19T02:03:36.607 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002694288 2024-04-19T02:03:36.607 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.607 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.607 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:36.607 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.700773+0000", 2024-04-19T02:03:36.608 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0026718940000000002 2024-04-19T02:03:36.608 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.608 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.608 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.608 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.700903+0000", 2024-04-19T02:03:36.608 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013001199999999999 2024-04-19T02:03:36.608 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.608 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.608 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.608 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.608 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.608 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.4s2 42/26 ECSubWrite(tid=43, reqid=osd.1.0:43, at_version=42'30, trim_to=0'0, roll_forward_to=40'28))", 2024-04-19T02:03:36.608 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.738323+0000", 2024-04-19T02:03:36.608 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.8644156180000007, 2024-04-19T02:03:36.608 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0089071480000000001, 2024-04-19T02:03:36.609 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.609 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.609 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.609 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.609 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.609 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.609 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.738323+0000", 2024-04-19T02:03:36.609 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.609 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.609 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.609 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.609 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.738323+0000", 2024-04-19T02:03:36.609 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.609 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.609 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.610 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.610 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.738323+0000", 2024-04-19T02:03:36.610 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0699999999999996e-07 2024-04-19T02:03:36.610 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.610 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.610 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.610 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.738379+0000", 2024-04-19T02:03:36.610 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5263000000000003e-05 2024-04-19T02:03:36.610 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.610 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.610 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.610 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.738410+0000", 2024-04-19T02:03:36.610 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1744e-05 2024-04-19T02:03:36.610 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.611 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.611 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.611 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.738437+0000", 2024-04-19T02:03:36.611 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6378000000000001e-05 2024-04-19T02:03:36.611 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.611 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.611 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.611 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.738669+0000", 2024-04-19T02:03:36.611 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00023269799999999999 2024-04-19T02:03:36.611 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.611 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.611 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.611 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.738698+0000", 2024-04-19T02:03:36.611 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8260999999999999e-05 2024-04-19T02:03:36.612 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.612 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.612 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.612 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747031+0000", 2024-04-19T02:03:36.612 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0083331759999999994 2024-04-19T02:03:36.612 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.612 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.612 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.612 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747230+0000", 2024-04-19T02:03:36.612 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00019892099999999999 2024-04-19T02:03:36.612 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.612 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.612 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.612 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.613 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.613 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.4s2 42/26 ECSubWrite(tid=44, reqid=osd.1.0:44, at_version=42'32, trim_to=0'0, roll_forward_to=40'28))", 2024-04-19T02:03:36.613 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.739493+0000", 2024-04-19T02:03:36.613 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.8632449019999999, 2024-04-19T02:03:36.613 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015165089, 2024-04-19T02:03:36.613 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.613 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.613 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.613 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.613 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.613 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.613 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.739493+0000", 2024-04-19T02:03:36.613 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.613 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.613 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.614 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.614 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.739493+0000", 2024-04-19T02:03:36.614 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.614 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.614 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.614 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.614 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.739495+0000", 2024-04-19T02:03:36.614 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8729999999999999e-06 2024-04-19T02:03:36.614 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.614 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.614 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.614 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.739550+0000", 2024-04-19T02:03:36.614 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5080000000000001e-05 2024-04-19T02:03:36.614 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.615 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.615 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.615 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.739584+0000", 2024-04-19T02:03:36.615 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3396999999999999e-05 2024-04-19T02:03:36.615 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.615 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.615 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.615 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.739611+0000", 2024-04-19T02:03:36.615 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7733999999999999e-05 2024-04-19T02:03:36.615 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.615 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.615 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.615 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.740435+0000", 2024-04-19T02:03:36.615 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00082331600000000004 2024-04-19T02:03:36.615 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.616 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.616 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.616 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.740463+0000", 2024-04-19T02:03:36.616 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7991999999999999e-05 2024-04-19T02:03:36.616 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.616 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.616 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.616 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.754535+0000", 2024-04-19T02:03:36.616 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014072092 2024-04-19T02:03:36.616 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.616 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.616 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.616 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.754658+0000", 2024-04-19T02:03:36.616 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012360499999999999 2024-04-19T02:03:36.616 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.616 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.617 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.617 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.617 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.617 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.8s7 42/26 ECSubWrite(tid=71, reqid=osd.4.0:71, at_version=42'33, trim_to=0'0, roll_forward_to=37'31))", 2024-04-19T02:03:36.617 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.747600+0000", 2024-04-19T02:03:36.617 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.8551377989999995, 2024-04-19T02:03:36.617 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012073659, 2024-04-19T02:03:36.617 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.617 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.617 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.617 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.617 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.617 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.617 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747600+0000", 2024-04-19T02:03:36.617 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.618 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.618 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.618 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.618 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747600+0000", 2024-04-19T02:03:36.618 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.618 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.618 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.618 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.618 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747601+0000", 2024-04-19T02:03:36.618 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5400000000000001e-07 2024-04-19T02:03:36.618 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.618 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.618 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.619 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747647+0000", 2024-04-19T02:03:36.619 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6529000000000001e-05 2024-04-19T02:03:36.619 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.619 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.619 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.619 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747671+0000", 2024-04-19T02:03:36.619 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3425e-05 2024-04-19T02:03:36.619 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.619 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.619 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.619 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747702+0000", 2024-04-19T02:03:36.619 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1531e-05 2024-04-19T02:03:36.619 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.619 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.619 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.619 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749865+0000", 2024-04-19T02:03:36.620 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0021629150000000001 2024-04-19T02:03:36.620 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.620 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.620 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.620 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749891+0000", 2024-04-19T02:03:36.620 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5568e-05 2024-04-19T02:03:36.620 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.620 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.620 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.620 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759578+0000", 2024-04-19T02:03:36.620 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0096872139999999995 2024-04-19T02:03:36.620 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.620 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.620 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.620 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759674+0000", 2024-04-19T02:03:36.621 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5922999999999996e-05 2024-04-19T02:03:36.621 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.621 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.621 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.621 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.621 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.621 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.bs5 42/26 ECSubWrite(tid=133, reqid=osd.3.0:133, at_version=42'21, trim_to=0'0, roll_forward_to=39'19))", 2024-04-19T02:03:36.621 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.748033+0000", 2024-04-19T02:03:36.621 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.8547053390000006, 2024-04-19T02:03:36.621 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012077704999999999, 2024-04-19T02:03:36.621 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.621 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.621 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.621 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.622 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.622 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.622 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748033+0000", 2024-04-19T02:03:36.622 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.622 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.622 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.622 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.622 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748033+0000", 2024-04-19T02:03:36.622 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.622 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.622 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.622 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.622 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748035+0000", 2024-04-19T02:03:36.622 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.75e-06 2024-04-19T02:03:36.622 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.623 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.623 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.623 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748087+0000", 2024-04-19T02:03:36.623 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2456000000000001e-05 2024-04-19T02:03:36.623 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.623 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.623 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.623 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748118+0000", 2024-04-19T02:03:36.623 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0728000000000001e-05 2024-04-19T02:03:36.623 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.623 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.623 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.623 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748141+0000", 2024-04-19T02:03:36.623 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2824e-05 2024-04-19T02:03:36.623 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.624 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.624 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.624 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748342+0000", 2024-04-19T02:03:36.624 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000201444 2024-04-19T02:03:36.624 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.624 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.624 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.624 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748366+0000", 2024-04-19T02:03:36.624 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4278999999999999e-05 2024-04-19T02:03:36.624 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.624 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.624 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.624 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760028+0000", 2024-04-19T02:03:36.624 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011662077 2024-04-19T02:03:36.625 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.625 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.625 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.625 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760111+0000", 2024-04-19T02:03:36.625 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2146999999999998e-05 2024-04-19T02:03:36.625 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.625 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.625 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.625 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.625 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.625 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.bs5 42/26 ECSubWrite(tid=135, reqid=osd.3.0:135, at_version=42'23, trim_to=0'0, roll_forward_to=39'19))", 2024-04-19T02:03:36.625 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.749533+0000", 2024-04-19T02:03:36.625 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.8532047889999994, 2024-04-19T02:03:36.625 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012068692000000001, 2024-04-19T02:03:36.625 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.626 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.626 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.626 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.626 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.626 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.626 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749533+0000", 2024-04-19T02:03:36.626 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.626 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.626 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.626 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.626 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749533+0000", 2024-04-19T02:03:36.626 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.626 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.626 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.626 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.627 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749535+0000", 2024-04-19T02:03:36.627 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7999999999999999e-06 2024-04-19T02:03:36.627 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.627 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.627 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.627 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749586+0000", 2024-04-19T02:03:36.627 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1202999999999997e-05 2024-04-19T02:03:36.627 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.627 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.627 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.627 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749618+0000", 2024-04-19T02:03:36.627 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1393999999999998e-05 2024-04-19T02:03:36.627 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.627 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.627 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.628 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749817+0000", 2024-04-19T02:03:36.628 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00019913799999999999 2024-04-19T02:03:36.628 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.628 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.628 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.628 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.757176+0000", 2024-04-19T02:03:36.628 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0073591949999999998 2024-04-19T02:03:36.628 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.628 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.628 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.628 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.757200+0000", 2024-04-19T02:03:36.628 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4388000000000001e-05 2024-04-19T02:03:36.628 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.628 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.628 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.629 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761527+0000", 2024-04-19T02:03:36.629 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0043261899999999997 2024-04-19T02:03:36.629 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.629 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.629 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.629 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761602+0000", 2024-04-19T02:03:36.629 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5383999999999995e-05 2024-04-19T02:03:36.629 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.629 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.629 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.629 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.629 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.629 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 42/26 ECSubWriteReply(tid=51, last_complete=42'27, committed=1, applied=1))", 2024-04-19T02:03:36.629 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.753536+0000", 2024-04-19T02:03:36.629 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.8492022059999993, 2024-04-19T02:03:36.630 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0092102, 2024-04-19T02:03:36.630 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.630 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.630 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.630 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.630 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.630 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.630 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753536+0000", 2024-04-19T02:03:36.630 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.630 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.630 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.630 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.630 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753536+0000", 2024-04-19T02:03:36.630 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.631 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.631 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.631 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.631 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753538+0000", 2024-04-19T02:03:36.631 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.863e-06 2024-04-19T02:03:36.631 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.631 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.631 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.631 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753550+0000", 2024-04-19T02:03:36.631 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2102e-05 2024-04-19T02:03:36.631 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.631 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.631 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.631 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753580+0000", 2024-04-19T02:03:36.631 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0343999999999999e-05 2024-04-19T02:03:36.632 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.632 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.632 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.632 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753601+0000", 2024-04-19T02:03:36.632 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0499000000000001e-05 2024-04-19T02:03:36.632 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.632 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.632 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.632 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.762682+0000", 2024-04-19T02:03:36.632 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0090811969999999992 2024-04-19T02:03:36.632 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.632 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.632 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.632 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.762746+0000", 2024-04-19T02:03:36.632 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4195000000000005e-05 2024-04-19T02:03:36.633 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.633 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.633 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.633 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.633 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.633 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 42/26 ECSubWriteReply(tid=51, last_complete=42'27, committed=1, applied=1))", 2024-04-19T02:03:36.633 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.753740+0000", 2024-04-19T02:03:36.633 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.8489986389999995, 2024-04-19T02:03:36.633 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0091385830000000005, 2024-04-19T02:03:36.633 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.633 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.633 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.633 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.633 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.633 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.634 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753740+0000", 2024-04-19T02:03:36.634 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.634 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.634 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.634 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.634 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753740+0000", 2024-04-19T02:03:36.634 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.634 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.634 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.634 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.634 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753741+0000", 2024-04-19T02:03:36.634 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3030000000000001e-06 2024-04-19T02:03:36.634 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.634 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.634 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.635 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753754+0000", 2024-04-19T02:03:36.635 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3079e-05 2024-04-19T02:03:36.635 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.635 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.635 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.635 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753777+0000", 2024-04-19T02:03:36.635 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2841999999999999e-05 2024-04-19T02:03:36.635 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.635 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.635 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.635 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753796+0000", 2024-04-19T02:03:36.635 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9445000000000001e-05 2024-04-19T02:03:36.635 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.635 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.635 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.636 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.762824+0000", 2024-04-19T02:03:36.636 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0090278279999999999 2024-04-19T02:03:36.636 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.636 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.636 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.636 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.762878+0000", 2024-04-19T02:03:36.636 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4085999999999997e-05 2024-04-19T02:03:36.636 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.636 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.636 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.636 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.636 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.636 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.4s2 46/26 ECSubWrite(tid=58, reqid=osd.1.0:58, at_version=46'44, trim_to=0'0, roll_forward_to=45'42))", 2024-04-19T02:03:36.636 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:30.060651+0000", 2024-04-19T02:03:36.636 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 6.5420872030000004, 2024-04-19T02:03:36.637 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0085905930000000005, 2024-04-19T02:03:36.637 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.637 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.637 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.637 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.637 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.637 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.637 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.060651+0000", 2024-04-19T02:03:36.637 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.637 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.637 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.637 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.637 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.060651+0000", 2024-04-19T02:03:36.637 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.637 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.638 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.638 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.638 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.060651+0000", 2024-04-19T02:03:36.638 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6899999999999999e-07 2024-04-19T02:03:36.638 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.638 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.638 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.638 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.060673+0000", 2024-04-19T02:03:36.638 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1917999999999999e-05 2024-04-19T02:03:36.638 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.638 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.638 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.638 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.060694+0000", 2024-04-19T02:03:36.638 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0962999999999999e-05 2024-04-19T02:03:36.639 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.639 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.639 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.639 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.060705+0000", 2024-04-19T02:03:36.639 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1090999999999999e-05 2024-04-19T02:03:36.639 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.639 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.639 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.639 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.060909+0000", 2024-04-19T02:03:36.639 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000203795 2024-04-19T02:03:36.639 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.639 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.639 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.639 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.060919+0000", 2024-04-19T02:03:36.639 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6020000000000006e-06 2024-04-19T02:03:36.640 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.640 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.640 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.640 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.069154+0000", 2024-04-19T02:03:36.640 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0082356979999999996 2024-04-19T02:03:36.640 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.640 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.640 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.640 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.069242+0000", 2024-04-19T02:03:36.640 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7256999999999995e-05 2024-04-19T02:03:36.640 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.640 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.640 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.640 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.640 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.641 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s2 46/26 ECSubWrite(tid=171, reqid=client.4509.0:422, at_version=46'85, trim_to=0'0, roll_forward_to=46'75))", 2024-04-19T02:03:36.641 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:30.225060+0000", 2024-04-19T02:03:36.641 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 6.3776782260000004, 2024-04-19T02:03:36.641 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0083378600000000008, 2024-04-19T02:03:36.641 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.641 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.641 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.641 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.641 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.641 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.641 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.225060+0000", 2024-04-19T02:03:36.641 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.641 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.641 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.641 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.642 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.225060+0000", 2024-04-19T02:03:36.642 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.642 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.642 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.642 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.642 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.225061+0000", 2024-04-19T02:03:36.642 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2619999999999999e-06 2024-04-19T02:03:36.642 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.642 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.642 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.642 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.226077+0000", 2024-04-19T02:03:36.642 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0010156430000000001 2024-04-19T02:03:36.642 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.642 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.643 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.643 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.226090+0000", 2024-04-19T02:03:36.643 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2975e-05 2024-04-19T02:03:36.643 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.643 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.643 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.643 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.226098+0000", 2024-04-19T02:03:36.643 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4419999999999998e-06 2024-04-19T02:03:36.643 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.643 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.643 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.644 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.226345+0000", 2024-04-19T02:03:36.644 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00024690999999999998 2024-04-19T02:03:36.644 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.644 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.644 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.644 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.226368+0000", 2024-04-19T02:03:36.644 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3161999999999999e-05 2024-04-19T02:03:36.644 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.644 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.644 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.644 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.233358+0000", 2024-04-19T02:03:36.644 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0069900689999999998 2024-04-19T02:03:36.644 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.644 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.645 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.645 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.233398+0000", 2024-04-19T02:03:36.645 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9397000000000002e-05 2024-04-19T02:03:36.645 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.645 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.645 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.645 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.645 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.645 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.1s4 47/26 ECSubWrite(tid=29, reqid=osd.0.0:29, at_version=47'15, trim_to=0'0, roll_forward_to=46'13))", 2024-04-19T02:03:36.645 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.087131+0000", 2024-04-19T02:03:36.645 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 5.5156070079999999, 2024-04-19T02:03:36.645 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.028478961000000001, 2024-04-19T02:03:36.645 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.645 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.645 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.646 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.646 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.646 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.646 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087131+0000", 2024-04-19T02:03:36.646 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.646 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.646 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.646 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.646 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087131+0000", 2024-04-19T02:03:36.646 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.646 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.646 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.646 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.646 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087132+0000", 2024-04-19T02:03:36.647 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4000000000000001e-07 2024-04-19T02:03:36.647 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.647 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.647 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.647 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087185+0000", 2024-04-19T02:03:36.647 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3016999999999997e-05 2024-04-19T02:03:36.647 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.647 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.647 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.647 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087232+0000", 2024-04-19T02:03:36.647 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6989e-05 2024-04-19T02:03:36.647 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.647 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.647 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.648 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087259+0000", 2024-04-19T02:03:36.648 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7199000000000002e-05 2024-04-19T02:03:36.648 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.648 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.648 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.648 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087431+0000", 2024-04-19T02:03:36.648 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017193199999999999 2024-04-19T02:03:36.648 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.648 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.648 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.648 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087457+0000", 2024-04-19T02:03:36.648 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6148000000000001e-05 2024-04-19T02:03:36.648 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.648 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.648 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.649 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.115506+0000", 2024-04-19T02:03:36.649 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.028048812999999999 2024-04-19T02:03:36.649 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.649 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.649 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.649 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.115610+0000", 2024-04-19T02:03:36.649 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010412300000000001 2024-04-19T02:03:36.649 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.649 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.649 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.649 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.649 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.649 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.1s4 47/26 ECSubWrite(tid=30, reqid=osd.0.0:30, at_version=47'17, trim_to=0'0, roll_forward_to=46'13))", 2024-04-19T02:03:36.649 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.087466+0000", 2024-04-19T02:03:36.650 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 5.5152726210000003, 2024-04-19T02:03:36.650 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.030894781, 2024-04-19T02:03:36.650 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.650 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.650 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.650 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.650 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.650 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.650 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087466+0000", 2024-04-19T02:03:36.650 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.650 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.650 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.650 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.650 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087466+0000", 2024-04-19T02:03:36.650 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.651 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.651 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.651 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.651 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087466+0000", 2024-04-19T02:03:36.651 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0400000000000005e-07 2024-04-19T02:03:36.651 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.651 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.651 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.651 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087524+0000", 2024-04-19T02:03:36.651 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7632e-05 2024-04-19T02:03:36.651 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.651 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.651 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.651 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087550+0000", 2024-04-19T02:03:36.652 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5871999999999999e-05 2024-04-19T02:03:36.652 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.652 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.652 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.652 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087570+0000", 2024-04-19T02:03:36.652 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0531999999999999e-05 2024-04-19T02:03:36.652 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.652 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.652 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.652 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.091486+0000", 2024-04-19T02:03:36.652 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039151009999999998 2024-04-19T02:03:36.652 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.652 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.652 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.652 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.091519+0000", 2024-04-19T02:03:36.653 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3874999999999997e-05 2024-04-19T02:03:36.653 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.653 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.653 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.653 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.118273+0000", 2024-04-19T02:03:36.653 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.026753835 2024-04-19T02:03:36.653 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.653 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.653 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.653 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.118360+0000", 2024-04-19T02:03:36.653 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7029999999999999e-05 2024-04-19T02:03:36.653 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.653 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.653 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.653 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.653 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.654 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s2 47/26 ECSubWrite(tid=181, reqid=osd.3.0:181, at_version=47'87, trim_to=0'0, roll_forward_to=46'85))", 2024-04-19T02:03:36.654 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.088687+0000", 2024-04-19T02:03:36.654 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 5.5140512370000003, 2024-04-19T02:03:36.654 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.027025486000000001, 2024-04-19T02:03:36.654 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.654 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.654 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.654 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.654 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.654 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.654 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088687+0000", 2024-04-19T02:03:36.654 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.654 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.654 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.655 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.655 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088687+0000", 2024-04-19T02:03:36.655 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.655 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.655 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.655 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.655 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088687+0000", 2024-04-19T02:03:36.655 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7999999999999996e-07 2024-04-19T02:03:36.655 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.655 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.655 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.655 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088786+0000", 2024-04-19T02:03:36.655 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8510000000000004e-05 2024-04-19T02:03:36.655 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.655 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.656 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.656 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088822+0000", 2024-04-19T02:03:36.656 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6284e-05 2024-04-19T02:03:36.656 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.656 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.656 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.656 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088846+0000", 2024-04-19T02:03:36.656 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4125999999999999e-05 2024-04-19T02:03:36.656 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.656 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.656 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.656 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089107+0000", 2024-04-19T02:03:36.656 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00026044799999999998 2024-04-19T02:03:36.656 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.657 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.657 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.657 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089134+0000", 2024-04-19T02:03:36.657 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7067999999999999e-05 2024-04-19T02:03:36.657 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.657 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.657 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.657 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.115620+0000", 2024-04-19T02:03:36.657 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.026486464000000001 2024-04-19T02:03:36.657 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.657 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.657 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.657 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.115712+0000", 2024-04-19T02:03:36.657 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2106000000000002e-05 2024-04-19T02:03:36.657 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.658 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.658 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.658 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.658 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.658 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.bs5 47/26 ECSubWrite(tid=180, reqid=osd.3.0:180, at_version=47'36, trim_to=0'0, roll_forward_to=46'34))", 2024-04-19T02:03:36.658 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.088907+0000", 2024-04-19T02:03:36.658 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 5.5138316280000002, 2024-04-19T02:03:36.658 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.028441793999999999, 2024-04-19T02:03:36.658 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.658 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.658 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.658 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.659 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.659 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.659 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088907+0000", 2024-04-19T02:03:36.659 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.659 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.659 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.659 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.659 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088907+0000", 2024-04-19T02:03:36.659 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.659 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.659 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.659 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.659 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088908+0000", 2024-04-19T02:03:36.659 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0100000000000001e-06 2024-04-19T02:03:36.660 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.660 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.660 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.660 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088952+0000", 2024-04-19T02:03:36.660 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4613000000000001e-05 2024-04-19T02:03:36.660 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.660 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.660 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.660 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088975+0000", 2024-04-19T02:03:36.660 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2518999999999999e-05 2024-04-19T02:03:36.660 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.660 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.660 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.660 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088993+0000", 2024-04-19T02:03:36.661 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7855000000000001e-05 2024-04-19T02:03:36.661 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.661 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.661 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.661 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089130+0000", 2024-04-19T02:03:36.661 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013709099999999999 2024-04-19T02:03:36.661 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.661 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.661 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.661 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089155+0000", 2024-04-19T02:03:36.661 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4924999999999999e-05 2024-04-19T02:03:36.661 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.661 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.661 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.661 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.117263+0000", 2024-04-19T02:03:36.661 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.028107953000000001 2024-04-19T02:03:36.662 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.662 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.662 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.662 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.117348+0000", 2024-04-19T02:03:36.662 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5828000000000006e-05 2024-04-19T02:03:36.662 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.662 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.662 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.662 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.662 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.662 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s2 47/26 ECSubWrite(tid=182, reqid=osd.3.0:182, at_version=47'89, trim_to=0'0, roll_forward_to=46'85))", 2024-04-19T02:03:36.662 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.089036+0000", 2024-04-19T02:03:36.662 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 5.5137026000000002, 2024-04-19T02:03:36.662 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.027692491, 2024-04-19T02:03:36.663 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.663 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.663 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.663 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.663 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.663 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.663 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089036+0000", 2024-04-19T02:03:36.663 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.663 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.663 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.663 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.663 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089036+0000", 2024-04-19T02:03:36.663 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.663 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.663 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.664 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.664 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089036+0000", 2024-04-19T02:03:36.664 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7599999999999996e-07 2024-04-19T02:03:36.664 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.664 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.664 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.664 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089097+0000", 2024-04-19T02:03:36.664 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0223000000000003e-05 2024-04-19T02:03:36.664 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.664 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.664 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.664 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089117+0000", 2024-04-19T02:03:36.664 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0724999999999999e-05 2024-04-19T02:03:36.664 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.664 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.665 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.665 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089134+0000", 2024-04-19T02:03:36.665 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7065e-05 2024-04-19T02:03:36.665 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.665 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.665 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.665 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.091684+0000", 2024-04-19T02:03:36.665 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0025499780000000001 2024-04-19T02:03:36.665 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.665 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.665 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.665 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.091723+0000", 2024-04-19T02:03:36.665 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8787e-05 2024-04-19T02:03:36.665 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.666 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.666 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.666 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.116611+0000", 2024-04-19T02:03:36.666 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.024888321000000001 2024-04-19T02:03:36.666 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.666 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.666 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.666 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.116728+0000", 2024-04-19T02:03:36.666 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000116616 2024-04-19T02:03:36.666 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.666 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.666 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.666 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.666 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.667 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.bs5 47/26 ECSubWrite(tid=183, reqid=osd.3.0:183, at_version=47'38, trim_to=0'0, roll_forward_to=46'34))", 2024-04-19T02:03:36.667 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.089162+0000", 2024-04-19T02:03:36.667 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 5.5135762330000002, 2024-04-19T02:03:36.667 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.031704567000000003, 2024-04-19T02:03:36.667 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.667 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.667 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.667 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.667 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.667 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.667 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089162+0000", 2024-04-19T02:03:36.667 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.667 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.667 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.667 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.668 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089162+0000", 2024-04-19T02:03:36.668 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.668 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.668 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.668 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.668 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089163+0000", 2024-04-19T02:03:36.668 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8999999999999996e-07 2024-04-19T02:03:36.668 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.668 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.668 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.668 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089208+0000", 2024-04-19T02:03:36.668 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5670999999999997e-05 2024-04-19T02:03:36.668 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.668 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.668 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.669 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089231+0000", 2024-04-19T02:03:36.669 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2898999999999999e-05 2024-04-19T02:03:36.669 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.669 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.669 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.669 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089248+0000", 2024-04-19T02:03:36.669 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7170999999999999e-05 2024-04-19T02:03:36.669 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.669 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.669 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.669 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.091699+0000", 2024-04-19T02:03:36.669 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0024503010000000002 2024-04-19T02:03:36.669 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.669 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.669 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.670 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.091731+0000", 2024-04-19T02:03:36.670 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2301000000000001e-05 2024-04-19T02:03:36.670 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.670 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.670 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.670 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.120784+0000", 2024-04-19T02:03:36.670 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.029052923000000001 2024-04-19T02:03:36.670 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.670 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.670 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.670 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.120866+0000", 2024-04-19T02:03:36.670 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2610999999999999e-05 2024-04-19T02:03:36.670 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.670 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.671 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.671 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.671 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.671 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.8s7 47/26 ECSubWrite(tid=86, reqid=osd.4.0:86, at_version=47'44, trim_to=0'0, roll_forward_to=46'42))", 2024-04-19T02:03:36.671 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.092233+0000", 2024-04-19T02:03:36.671 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 5.5105056169999997, 2024-04-19T02:03:36.671 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.026209364999999998, 2024-04-19T02:03:36.671 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.671 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.671 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:36.671 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.671 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.671 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.671 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092233+0000", 2024-04-19T02:03:36.672 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.672 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.672 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.672 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.672 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092233+0000", 2024-04-19T02:03:36.672 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.672 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.672 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.672 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.672 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092233+0000", 2024-04-19T02:03:36.672 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8699999999999995e-07 2024-04-19T02:03:36.672 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.672 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.672 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.673 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092282+0000", 2024-04-19T02:03:36.673 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8869999999999998e-05 2024-04-19T02:03:36.673 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.673 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.673 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.673 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092311+0000", 2024-04-19T02:03:36.673 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9201e-05 2024-04-19T02:03:36.673 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.673 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.673 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.673 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092342+0000", 2024-04-19T02:03:36.673 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1275000000000001e-05 2024-04-19T02:03:36.673 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.673 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.673 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.674 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092597+0000", 2024-04-19T02:03:36.674 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00025494899999999997 2024-04-19T02:03:36.674 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.674 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.674 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:36.674 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092622+0000", 2024-04-19T02:03:36.674 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4355e-05 2024-04-19T02:03:36.674 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.674 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.674 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:36.674 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.118397+0000", 2024-04-19T02:03:36.674 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.025775005 2024-04-19T02:03:36.674 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.674 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.118442+0000", 2024-04-19T02:03:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5123000000000001e-05 2024-04-19T02:03:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:461 3.3s0 3:c26f510a:::smithi05930959-46:head [read 0~770048,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:03:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.138495+0000", 2024-04-19T02:03:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 5.4642432860000003, 2024-04-19T02:03:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010087319000000001, 2024-04-19T02:03:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:36.676 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:36.676 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:36.676 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:36.676 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 461 2024-04-19T02:03:36.676 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.676 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.676 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.676 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.676 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138495+0000", 2024-04-19T02:03:36.676 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.676 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.676 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.676 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.677 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138495+0000", 2024-04-19T02:03:36.677 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.677 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.677 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.677 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.677 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138498+0000", 2024-04-19T02:03:36.677 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2990000000000001e-06 2024-04-19T02:03:36.677 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.677 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.677 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.677 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138513+0000", 2024-04-19T02:03:36.677 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4379000000000001e-05 2024-04-19T02:03:36.677 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.677 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.677 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.678 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138543+0000", 2024-04-19T02:03:36.678 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0201000000000001e-05 2024-04-19T02:03:36.678 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.678 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.678 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.678 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138557+0000", 2024-04-19T02:03:36.678 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4056000000000001e-05 2024-04-19T02:03:36.678 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.678 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.678 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.678 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138665+0000", 2024-04-19T02:03:36.678 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010802300000000001 2024-04-19T02:03:36.678 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.678 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.679 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:36.679 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138814+0000", 2024-04-19T02:03:36.679 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014875899999999999 2024-04-19T02:03:36.679 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.679 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.679 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.679 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.148582+0000", 2024-04-19T02:03:36.679 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0097686019999999995 2024-04-19T02:03:36.679 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.679 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.679 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.679 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.679 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.679 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:462 3.3s0 3:c26f510a:::smithi05930959-46:head [read 0~770048,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:03:36.679 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.138590+0000", 2024-04-19T02:03:36.680 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 5.4641483700000002, 2024-04-19T02:03:36.680 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011711944, 2024-04-19T02:03:36.680 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.680 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.680 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:36.680 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:36.680 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:36.680 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:36.680 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 462 2024-04-19T02:03:36.680 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.680 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.680 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.680 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.680 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138590+0000", 2024-04-19T02:03:36.680 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.681 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.681 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.681 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.681 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138590+0000", 2024-04-19T02:03:36.681 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.681 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.681 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.681 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.681 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138591+0000", 2024-04-19T02:03:36.681 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1100000000000005e-07 2024-04-19T02:03:36.681 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.681 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.681 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.681 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138596+0000", 2024-04-19T02:03:36.681 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6740000000000002e-06 2024-04-19T02:03:36.682 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.682 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.682 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.682 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138611+0000", 2024-04-19T02:03:36.682 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4425e-05 2024-04-19T02:03:36.682 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.682 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.682 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.682 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138619+0000", 2024-04-19T02:03:36.682 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1880000000000002e-06 2024-04-19T02:03:36.682 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.682 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.682 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.682 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.139793+0000", 2024-04-19T02:03:36.683 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001174247 2024-04-19T02:03:36.683 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.683 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.683 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:36.683 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.140277+0000", 2024-04-19T02:03:36.683 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00048428099999999998 2024-04-19T02:03:36.683 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.683 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.683 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.683 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.150302+0000", 2024-04-19T02:03:36.683 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010024318000000001 2024-04-19T02:03:36.683 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.683 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.683 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.683 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.684 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.684 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:463 3.3s0 3:c26f510a:::smithi05930959-46:head [read 0~770048,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:03:36.684 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.138651+0000", 2024-04-19T02:03:36.684 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 5.4640874940000002, 2024-04-19T02:03:36.684 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012901299999999999, 2024-04-19T02:03:36.684 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:36.684 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:36.684 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:36.684 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:36.684 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:36.684 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:36.684 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 463 2024-04-19T02:03:36.684 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.684 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:36.684 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.685 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:36.685 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138651+0000", 2024-04-19T02:03:36.685 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.685 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.685 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.685 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:36.685 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138651+0000", 2024-04-19T02:03:36.685 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:36.685 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.685 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.685 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:36.685 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138651+0000", 2024-04-19T02:03:36.685 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4000000000000001e-07 2024-04-19T02:03:36.685 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.685 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.686 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:36.686 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138657+0000", 2024-04-19T02:03:36.686 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9479999999999996e-06 2024-04-19T02:03:36.686 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.686 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.686 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:36.686 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138667+0000", 2024-04-19T02:03:36.686 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8679999999999994e-06 2024-04-19T02:03:36.686 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.686 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.686 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:36.686 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138675+0000", 2024-04-19T02:03:36.686 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4989999999999997e-06 2024-04-19T02:03:36.686 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.686 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.687 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:36.687 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.141119+0000", 2024-04-19T02:03:36.687 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0024445019999999999 2024-04-19T02:03:36.687 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.687 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.687 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:36.687 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.141379+0000", 2024-04-19T02:03:36.687 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000259595 2024-04-19T02:03:36.687 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:36.687 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:36.687 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:36.687 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.151552+0000", 2024-04-19T02:03:36.687 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010173148 2024-04-19T02:03:36.687 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.688 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.688 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.688 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:36.688 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:36.688 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:36.689 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-04-19T02:03:36.689 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T02:03:36.690 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-04-19T02:03:36.690 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T02:03:36.691 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-19T02:03:36.691 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:36.686+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:36.694 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-04-19T02:03:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T02:03:36.710 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-04-19T02:03:36.710 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T02:03:36.731 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-04-19T02:03:36.732 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T02:03:36.752 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-04-19T02:03:36.752 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T02:03:36.773 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-04-19T02:03:36.773 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T02:03:36.790 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:36.790+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:36.794 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-04-19T02:03:36.794 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T02:03:36.816 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-04-19T02:03:36.817 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T02:03:36.817 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:36.818 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:03:36.818 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-04-19T02:03:36.818 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:36.832 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-19T02:03:36.838 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-04-19T02:03:36.839 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T02:03:36.861 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-04-19T02:03:36.862 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T02:03:36.882 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-04-19T02:03:36.882 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T02:03:36.891 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:36.890+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:36.989 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:36.989 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:36.991 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:36.990+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:37.002 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-19T02:03:37.039 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.10 to 1 (10000) 2024-04-19T02:03:37.044 INFO:tasks.rados.rados.0.smithi059.stdout:208: done (2 left) 2024-04-19T02:03:37.044 INFO:tasks.rados.rados.0.smithi059.stdout:209: read oid 43 snap 12 2024-04-19T02:03:37.044 INFO:tasks.rados.rados.0.smithi059.stdout:209: expect (ObjNum 74 snap 6 seq_num 74) 2024-04-19T02:03:37.044 INFO:tasks.rados.rados.0.smithi059.stdout:210: copy_from oid 42 from oid 13 current snap is 13 2024-04-19T02:03:37.044 INFO:tasks.rados.rados.0.smithi059.stdout:206: finishing write tid 1 to smithi05930959-25 2024-04-19T02:03:37.044 INFO:tasks.rados.rados.0.smithi059.stdout:206: finishing write tid 2 to smithi05930959-25 2024-04-19T02:03:37.044 INFO:tasks.rados.rados.0.smithi059.stdout:206: finishing write tid 3 to smithi05930959-25 2024-04-19T02:03:37.045 INFO:tasks.rados.rados.0.smithi059.stdout:206: finishing write tid 4 to smithi05930959-25 2024-04-19T02:03:37.045 INFO:tasks.rados.rados.0.smithi059.stdout:206: finishing write tid 5 to smithi05930959-25 2024-04-19T02:03:37.045 INFO:tasks.rados.rados.0.smithi059.stdout:206: finishing write tid 6 to smithi05930959-25 2024-04-19T02:03:37.045 INFO:tasks.rados.rados.0.smithi059.stdout:206: oid 25 updating version 0 to 35 2024-04-19T02:03:37.045 INFO:tasks.rados.rados.0.smithi059.stdout:206: oid 25 version 35 is already newer than 34 2024-04-19T02:03:37.045 INFO:tasks.rados.rados.0.smithi059.stdout:206: oid 25 version 35 is already newer than 32 2024-04-19T02:03:37.045 INFO:tasks.rados.rados.0.smithi059.stdout:206: oid 25 updating version 35 to 36 2024-04-19T02:03:37.045 INFO:tasks.rados.rados.0.smithi059.stdout:206: oid 25 version 36 is already newer than 33 2024-04-19T02:03:37.045 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 25 v 36 (ObjNum 101 snap 13 seq_num 101) dirty exists 2024-04-19T02:03:37.045 INFO:tasks.rados.rados.0.smithi059.stdout:206: left oid 25 (ObjNum 101 snap 13 seq_num 101) 2024-04-19T02:03:37.045 INFO:tasks.rados.rados.0.smithi059.stdout:207: finishing copy_from racing read to smithi05930959-1 2024-04-19T02:03:37.045 INFO:tasks.rados.rados.0.smithi059.stdout:207: finishing copy_from to smithi05930959-1 2024-04-19T02:03:37.045 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 1 v 31 (ObjNum 71 snap 5 seq_num 71) dirty exists 2024-04-19T02:03:37.045 INFO:tasks.rados.rados.0.smithi059.stdout:206: done (3 left) 2024-04-19T02:03:37.046 INFO:tasks.rados.rados.0.smithi059.stdout:207: done (2 left) 2024-04-19T02:03:37.046 INFO:tasks.rados.rados.0.smithi059.stdout:211: delete oid 14 current snap is 13 2024-04-19T02:03:37.067 INFO:tasks.rados.rados.0.smithi059.stdout:211: done (2 left) 2024-04-19T02:03:37.067 INFO:tasks.rados.rados.0.smithi059.stdout:212: snap_create 2024-04-19T02:03:37.078 INFO:tasks.rados.rados.0.smithi059.stdout:209: expect (ObjNum 74 snap 6 seq_num 74) 2024-04-19T02:03:37.079 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-04-19T02:03:37.112 INFO:tasks.rados.rados.0.smithi059.stdout:210: finishing copy_from racing read to smithi05930959-42 2024-04-19T02:03:37.113 INFO:tasks.rados.rados.0.smithi059.stdout:210: finishing copy_from to smithi05930959-42 2024-04-19T02:03:37.113 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 42 v 11 (ObjNum 56 snap 1 seq_num 56) dirty exists 2024-04-19T02:03:37.113 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:37.110+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:37.170 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:37.170 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:37.182 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-19T02:03:37.194 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:37.194+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:37.295 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:37.294+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:37.304 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:37.304 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:37.319 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-19T02:03:37.396 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:37.398+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:37.436 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:37.436 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:37.450 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-19T02:03:37.497 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:37.494+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:37.598 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:37.598+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:37.620 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:37.620 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:37.635 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-19T02:03:37.699 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:37.698+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:37.764 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:37.764 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:37.779 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-19T02:03:37.823 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:37.826+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:37.901 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:37.902+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:37.906 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:37.906 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:37.920 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-19T02:03:38.002 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:38.002+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:38.041 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.11 to 1 (10000) 2024-04-19T02:03:38.044 INFO:tasks.rados.rados.0.smithi059.stdout:209: done (2 left) 2024-04-19T02:03:38.044 INFO:tasks.rados.rados.0.smithi059.stdout:210: done (1 left) 2024-04-19T02:03:38.044 INFO:tasks.rados.rados.0.smithi059.stdout:212: done (0 left) 2024-04-19T02:03:38.044 INFO:tasks.rados.rados.0.smithi059.stdout:213: delete oid 27 current snap is 14 2024-04-19T02:03:38.056 INFO:tasks.rados.rados.0.smithi059.stdout:213: done (0 left) 2024-04-19T02:03:38.056 INFO:tasks.rados.rados.0.smithi059.stdout:214: copy_from oid 21 from oid 29 current snap is 14 2024-04-19T02:03:38.056 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 3 current snap is 14 2024-04-19T02:03:38.056 INFO:tasks.rados.rados.0.smithi059.stdout:215: seq_num 104 ranges {49152=589824,638976=540672,1179648=622592} 2024-04-19T02:03:38.060 INFO:tasks.rados.rados.0.smithi059.stdout:215: writing smithi05930959-3 from 49152 to 638976 tid 1 2024-04-19T02:03:38.063 INFO:tasks.rados.rados.0.smithi059.stdout:215: writing smithi05930959-3 from 638976 to 1179648 tid 2 2024-04-19T02:03:38.067 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-19T02:03:38.068 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T02:03:38.068 INFO:tasks.rados.rados.0.smithi059.stdout:215: writing smithi05930959-3 from 1179648 to 1802240 tid 3 2024-04-19T02:03:38.068 INFO:tasks.rados.rados.0.smithi059.stdout:216: snap_remove snap 12 2024-04-19T02:03:38.102 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:38.102 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:38.102 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:38.102+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:38.117 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-19T02:03:38.138 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-19T02:03:38.203 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:38.202+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:38.298 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:38.298 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:38.304 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:38.302+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:38.318 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-19T02:03:38.380 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:03:38.381 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-19T02:03:38.399 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-04-19T02:03:38.215120+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64438816,"num_objects":52,"num_object_clones":6,"num_object_copies":404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":2,"num_read":407,"num_read_kb":90692,"num_write":625,"num_write_kb":82906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":676,"ondisk_log_size":676,"up":146,"acting":146,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1132462080,"kb_used":476252,"kb_used_data":127004,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1131985828,"statfs":{"total":1159641169920,"available":1159153487872,"internally_reserved":0,"allocated":130052096,"data_stored":128732868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12550144,"num_objects":-5,"num_object_clones":-6,"num_object_copies":-40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5,"num_whiteouts":1,"num_read":46,"num_read_kb":44541,"num_write":82,"num_write_kb":18736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001317"},"pg_stats":[{"pgid":"3.f","version":"47'35","reported_seq":165,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.067421+0000","last_change":"2024-04-19T02:03:31.110932+0000","last_active":"2024-04-19T02:03:34.067421+0000","last_peered":"2024-04-19T02:03:34.067421+0000","last_clean":"2024-04-19T02:03:34.067421+0000","last_became_active":"2024-04-19T02:03:11.728136+0000","last_became_peered":"2024-04-19T02:03:11.728136+0000","last_unstale":"2024-04-19T02:03:34.067421+0000","last_undegraded":"2024-04-19T02:03:34.067421+0000","last_fullsized":"2024-04-19T02:03:34.067421+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:10.999100+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.02342029,"stat_sum":{"num_bytes":4816896,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":10857,"num_write":22,"num_write_kb":5712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,2,3,5,8,0,7],"acting":[11,4,2,3,5,8,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"3.e","version":"50'34","reported_seq":162,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.062028+0000","last_change":"2024-04-19T02:03:31.115711+0000","last_active":"2024-04-19T02:03:34.062028+0000","last_peered":"2024-04-19T02:03:34.062028+0000","last_clean":"2024-04-19T02:03:34.062028+0000","last_became_active":"2024-04-19T02:03:11.730135+0000","last_became_peered":"2024-04-19T02:03:11.730135+0000","last_unstale":"2024-04-19T02:03:34.062028+0000","last_undegraded":"2024-04-19T02:03:34.062028+0000","last_fullsized":"2024-04-19T02:03:34.062028+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:33.583038+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.017062461000000001,"stat_sum":{"num_bytes":2392064,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":5477,"num_write":15,"num_write_kb":3552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5,11,0,2,3,9],"acting":[7,4,5,11,0,2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"3.d","version":"50'49","reported_seq":183,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.104358+0000","last_change":"2024-04-19T02:03:31.100201+0000","last_active":"2024-04-19T02:03:34.104358+0000","last_peered":"2024-04-19T02:03:34.104358+0000","last_clean":"2024-04-19T02:03:34.104358+0000","last_became_active":"2024-04-19T02:03:11.632682+0000","last_became_peered":"2024-04-19T02:03:11.632682+0000","last_unstale":"2024-04-19T02:03:34.104358+0000","last_undegraded":"2024-04-19T02:03:34.104358+0000","last_fullsized":"2024-04-19T02:03:34.104358+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:16.173000+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.018754509999999999,"stat_sum":{"num_bytes":7880704,"num_objects":6,"num_object_clones":2,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":26,"num_read_kb":5818,"num_write":46,"num_write_kb":10128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7,4,5,3,9,0],"acting":[8,11,7,4,5,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"3.c","version":"50'28","reported_seq":165,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.056568+0000","last_change":"2024-04-19T02:03:31.094093+0000","last_active":"2024-04-19T02:03:34.056568+0000","last_peered":"2024-04-19T02:03:34.056568+0000","last_clean":"2024-04-19T02:03:34.056568+0000","last_became_active":"2024-04-19T02:03:11.731568+0000","last_became_peered":"2024-04-19T02:03:11.731568+0000","last_unstale":"2024-04-19T02:03:34.056568+0000","last_undegraded":"2024-04-19T02:03:34.056568+0000","last_fullsized":"2024-04-19T02:03:34.056568+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:33.581741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029319580000000001,"stat_sum":{"num_bytes":2473984,"num_objects":5,"num_object_clones":1,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":31,"num_read_kb":11050,"num_write":15,"num_write_kb":3920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,11,7,4,2,0,5],"acting":[9,3,11,7,4,2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"3.b","version":"47'38","reported_seq":166,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.058008+0000","last_change":"2024-04-19T02:03:31.121913+0000","last_active":"2024-04-19T02:03:34.058008+0000","last_peered":"2024-04-19T02:03:34.058008+0000","last_clean":"2024-04-19T02:03:34.058008+0000","last_became_active":"2024-04-19T02:03:11.677316+0000","last_became_peered":"2024-04-19T02:03:11.677316+0000","last_unstale":"2024-04-19T02:03:34.058008+0000","last_undegraded":"2024-04-19T02:03:34.058008+0000","last_fullsized":"2024-04-19T02:03:34.058008+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:30.430288+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.043083218,"stat_sum":{"num_bytes":2834432,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":22,"num_read_kb":6047,"num_write":17,"num_write_kb":5072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8,6,4,10,1,7],"acting":[3,0,8,6,4,10,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"3.a","version":"47'55","reported_seq":184,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.066597+0000","last_change":"2024-04-19T02:03:31.115131+0000","last_active":"2024-04-19T02:03:34.066597+0000","last_peered":"2024-04-19T02:03:34.066597+0000","last_clean":"2024-04-19T02:03:34.066597+0000","last_became_active":"2024-04-19T02:03:11.727996+0000","last_became_peered":"2024-04-19T02:03:11.727996+0000","last_unstale":"2024-04-19T02:03:34.066597+0000","last_undegraded":"2024-04-19T02:03:34.066597+0000","last_fullsized":"2024-04-19T02:03:34.066597+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:13.595764+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.030207358,"stat_sum":{"num_bytes":7864320,"num_objects":4,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":16131,"num_write":68,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,1,9,5,7,3,2],"acting":[11,8,1,9,5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"3.9","version":"47'23","reported_seq":148,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.047784+0000","last_change":"2024-04-19T02:03:31.100671+0000","last_active":"2024-04-19T02:03:34.047784+0000","last_peered":"2024-04-19T02:03:34.047784+0000","last_clean":"2024-04-19T02:03:34.047784+0000","last_became_active":"2024-04-19T02:03:11.729255+0000","last_became_peered":"2024-04-19T02:03:11.729255+0000","last_unstale":"2024-04-19T02:03:34.047784+0000","last_undegraded":"2024-04-19T02:03:34.047784+0000","last_fullsized":"2024-04-19T02:03:34.047784+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:14.014641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056021719999999999,"stat_sum":{"num_bytes":3833856,"num_objects":3,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":17,"num_write_kb":4480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,10,2,9,1,7,11],"acting":[4,3,10,2,9,1,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"3.8","version":"49'46","reported_seq":176,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.047628+0000","last_change":"2024-04-19T02:03:31.120264+0000","last_active":"2024-04-19T02:03:34.047628+0000","last_peered":"2024-04-19T02:03:34.047628+0000","last_clean":"2024-04-19T02:03:34.047628+0000","last_became_active":"2024-04-19T02:03:11.679384+0000","last_became_peered":"2024-04-19T02:03:11.679384+0000","last_unstale":"2024-04-19T02:03:34.047628+0000","last_undegraded":"2024-04-19T02:03:34.047628+0000","last_fullsized":"2024-04-19T02:03:34.047628+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:39.588360+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.032089586000000003,"stat_sum":{"num_bytes":6193152,"num_objects":4,"num_object_clones":1,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":10138,"num_write":23,"num_write_kb":6192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7,11,5,9,6,10],"acting":[4,1,7,11,5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"1.0","version":"21'137","reported_seq":245,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.049982+0000","last_change":"2024-04-19T02:02:59.974166+0000","last_active":"2024-04-19T02:03:34.049982+0000","last_peered":"2024-04-19T02:03:34.049982+0000","last_clean":"2024-04-19T02:03:34.049982+0000","last_became_active":"2024-04-19T02:02:59.972005+0000","last_became_peered":"2024-04-19T02:02:59.972005+0000","last_unstale":"2024-04-19T02:03:34.049982+0000","last_undegraded":"2024-04-19T02:03:34.049982+0000","last_fullsized":"2024-04-19T02:03:34.049982+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:02:58.486225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:02:58.486225+0000","last_clean_scrub_stamp":"2024-04-19T02:02:58.486225+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:20.275707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":66,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.066268+0000","last_change":"2024-04-19T02:03:03.026806+0000","last_active":"2024-04-19T02:03:34.066268+0000","last_peered":"2024-04-19T02:03:34.066268+0000","last_clean":"2024-04-19T02:03:34.066268+0000","last_became_active":"2024-04-19T02:03:03.026063+0000","last_became_peered":"2024-04-19T02:03:03.026063+0000","last_unstale":"2024-04-19T02:03:34.066268+0000","last_undegraded":"2024-04-19T02:03:34.066268+0000","last_fullsized":"2024-04-19T02:03:34.066268+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:19.673016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"50'98","reported_seq":239,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.091760+0000","last_change":"2024-04-19T02:03:31.132520+0000","last_active":"2024-04-19T02:03:34.091760+0000","last_peered":"2024-04-19T02:03:34.091760+0000","last_clean":"2024-04-19T02:03:34.091760+0000","last_became_active":"2024-04-19T02:03:11.630475+0000","last_became_peered":"2024-04-19T02:03:11.630475+0000","last_unstale":"2024-04-19T02:03:34.091760+0000","last_undegraded":"2024-04-19T02:03:34.091760+0000","last_fullsized":"2024-04-19T02:03:34.091760+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:12.351889+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.053768872000000002,"stat_sum":{"num_bytes":8650752,"num_objects":8,"num_object_clones":1,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":49,"num_read_kb":14362,"num_write":87,"num_write_kb":12640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10,4,8,1,11,9],"acting":[3,5,10,4,8,1,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"2.0","version":"0'0","reported_seq":66,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.049970+0000","last_change":"2024-04-19T02:03:02.561076+0000","last_active":"2024-04-19T02:03:34.049970+0000","last_peered":"2024-04-19T02:03:34.049970+0000","last_clean":"2024-04-19T02:03:34.049970+0000","last_became_active":"2024-04-19T02:03:02.560210+0000","last_became_peered":"2024-04-19T02:03:02.560210+0000","last_unstale":"2024-04-19T02:03:34.049970+0000","last_undegraded":"2024-04-19T02:03:34.049970+0000","last_fullsized":"2024-04-19T02:03:34.049970+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:16.900004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"47'21","reported_seq":155,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-19T02:03:35.101457+0000","last_change":"2024-04-19T02:03:35.102074+0000","last_active":"2024-04-19T02:03:35.101457+0000","last_peered":"2024-04-19T02:03:35.101457+0000","last_clean":"2024-04-19T02:03:35.101457+0000","last_became_active":"2024-04-19T02:03:11.633394+0000","last_became_peered":"2024-04-19T02:03:11.633394+0000","last_unstale":"2024-04-19T02:03:35.101457+0000","last_undegraded":"2024-04-19T02:03:35.101457+0000","last_fullsized":"2024-04-19T02:03:35.101457+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:30.509710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085909099999999998,"stat_sum":{"num_bytes":4096000,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":1350,"num_write":15,"num_write_kb":4528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,4,5,10,7,9,1],"acting":[0,11,4,5,10,7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"},{"start":"d","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":61,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.057488+0000","last_change":"2024-04-19T02:03:09.594947+0000","last_active":"2024-04-19T02:03:34.057488+0000","last_peered":"2024-04-19T02:03:34.057488+0000","last_clean":"2024-04-19T02:03:34.057488+0000","last_became_active":"2024-04-19T02:03:09.594072+0000","last_became_peered":"2024-04-19T02:03:09.594072+0000","last_unstale":"2024-04-19T02:03:34.057488+0000","last_undegraded":"2024-04-19T02:03:34.057488+0000","last_fullsized":"2024-04-19T02:03:34.057488+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:29.461650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":117,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T02:03:33.053493+0000","last_change":"2024-04-19T02:03:31.096841+0000","last_active":"2024-04-19T02:03:33.053493+0000","last_peered":"2024-04-19T02:03:33.053493+0000","last_clean":"2024-04-19T02:03:33.053493+0000","last_became_active":"2024-04-19T02:03:11.678078+0000","last_became_peered":"2024-04-19T02:03:11.678078+0000","last_unstale":"2024-04-19T02:03:33.053493+0000","last_undegraded":"2024-04-19T02:03:33.053493+0000","last_fullsized":"2024-04-19T02:03:33.053493+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:25.778027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039275129999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,9,1,6,4,0,10],"acting":[5,7,9,1,6,4,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"2.2","version":"0'0","reported_seq":66,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.050250+0000","last_change":"2024-04-19T02:03:02.560133+0000","last_active":"2024-04-19T02:03:34.050250+0000","last_peered":"2024-04-19T02:03:34.050250+0000","last_clean":"2024-04-19T02:03:34.050250+0000","last_became_active":"2024-04-19T02:03:02.559377+0000","last_became_peered":"2024-04-19T02:03:02.559377+0000","last_unstale":"2024-04-19T02:03:34.050250+0000","last_undegraded":"2024-04-19T02:03:34.050250+0000","last_fullsized":"2024-04-19T02:03:34.050250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:30.728400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"48'15","reported_seq":136,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T02:03:33.063555+0000","last_change":"2024-04-19T02:03:31.119565+0000","last_active":"2024-04-19T02:03:33.063555+0000","last_peered":"2024-04-19T02:03:33.063555+0000","last_clean":"2024-04-19T02:03:33.063555+0000","last_became_active":"2024-04-19T02:03:11.677099+0000","last_became_peered":"2024-04-19T02:03:11.677099+0000","last_unstale":"2024-04-19T02:03:33.063555+0000","last_undegraded":"2024-04-19T02:03:33.063555+0000","last_fullsized":"2024-04-19T02:03:33.063555+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:36.180333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026550929999999999,"stat_sum":{"num_bytes":2441216,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":2368,"num_write":49,"num_write_kb":2384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,0,3,4,7,9,6],"acting":[10,8,0,3,4,7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"3.5","version":"45'33","reported_seq":151,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T02:03:33.063294+0000","last_change":"2024-04-19T02:03:31.116943+0000","last_active":"2024-04-19T02:03:33.063294+0000","last_peered":"2024-04-19T02:03:33.063294+0000","last_clean":"2024-04-19T02:03:33.063294+0000","last_became_active":"2024-04-19T02:03:11.728949+0000","last_became_peered":"2024-04-19T02:03:11.728949+0000","last_unstale":"2024-04-19T02:03:33.063294+0000","last_undegraded":"2024-04-19T02:03:33.063294+0000","last_fullsized":"2024-04-19T02:03:33.063294+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:28.041143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.02327926,"stat_sum":{"num_bytes":2293760,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":36,"num_write_kb":3584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2,9,8,1,6,5],"acting":[10,3,2,9,8,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"2.4","version":"0'0","reported_seq":64,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T02:03:33.053292+0000","last_change":"2024-04-19T02:03:02.597582+0000","last_active":"2024-04-19T02:03:33.053292+0000","last_peered":"2024-04-19T02:03:33.053292+0000","last_clean":"2024-04-19T02:03:33.053292+0000","last_became_active":"2024-04-19T02:03:02.597118+0000","last_became_peered":"2024-04-19T02:03:02.597118+0000","last_unstale":"2024-04-19T02:03:33.053292+0000","last_undegraded":"2024-04-19T02:03:33.053292+0000","last_fullsized":"2024-04-19T02:03:33.053292+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:13.683143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"47'48","reported_seq":174,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T02:03:33.053383+0000","last_change":"2024-04-19T02:03:31.086845+0000","last_active":"2024-04-19T02:03:33.053383+0000","last_peered":"2024-04-19T02:03:33.053383+0000","last_clean":"2024-04-19T02:03:33.053383+0000","last_became_active":"2024-04-19T02:03:11.730165+0000","last_became_peered":"2024-04-19T02:03:11.730165+0000","last_unstale":"2024-04-19T02:03:33.053383+0000","last_undegraded":"2024-04-19T02:03:33.053383+0000","last_fullsized":"2024-04-19T02:03:33.053383+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:14.215941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010297920000000001,"stat_sum":{"num_bytes":5734400,"num_objects":5,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":23,"num_read_kb":6442,"num_write":26,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,10,8,7,2,11,0],"acting":[1,5,10,8,7,2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"2.5","version":"0'0","reported_seq":66,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.062019+0000","last_change":"2024-04-19T02:03:02.595929+0000","last_active":"2024-04-19T02:03:34.062019+0000","last_peered":"2024-04-19T02:03:34.062019+0000","last_clean":"2024-04-19T02:03:34.062019+0000","last_became_active":"2024-04-19T02:03:02.595232+0000","last_became_peered":"2024-04-19T02:03:02.595232+0000","last_unstale":"2024-04-19T02:03:34.062019+0000","last_undegraded":"2024-04-19T02:03:34.062019+0000","last_fullsized":"2024-04-19T02:03:34.062019+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:04.077370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"46'14","reported_seq":135,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.058776+0000","last_change":"2024-04-19T02:03:31.122442+0000","last_active":"2024-04-19T02:03:34.058776+0000","last_peered":"2024-04-19T02:03:34.058776+0000","last_clean":"2024-04-19T02:03:34.058776+0000","last_became_active":"2024-04-19T02:03:11.675056+0000","last_became_peered":"2024-04-19T02:03:11.675056+0000","last_unstale":"2024-04-19T02:03:34.058776+0000","last_undegraded":"2024-04-19T02:03:34.058776+0000","last_fullsized":"2024-04-19T02:03:34.058776+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:12.521007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.043351122999999998,"stat_sum":{"num_bytes":2162688,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":6,"num_write_kb":2496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,9,0,7,5,11,1],"acting":[3,6,9,0,7,5,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"2.6","version":"0'0","reported_seq":64,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T02:03:33.053459+0000","last_change":"2024-04-19T02:03:02.988302+0000","last_active":"2024-04-19T02:03:33.053459+0000","last_peered":"2024-04-19T02:03:33.053459+0000","last_clean":"2024-04-19T02:03:33.053459+0000","last_became_active":"2024-04-19T02:03:02.988073+0000","last_became_peered":"2024-04-19T02:03:02.988073+0000","last_unstale":"2024-04-19T02:03:33.053459+0000","last_undegraded":"2024-04-19T02:03:33.053459+0000","last_fullsized":"2024-04-19T02:03:33.053459+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:28.455271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"29'2","reported_seq":133,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-19T02:03:35.102201+0000","last_change":"2024-04-19T02:03:35.102794+0000","last_active":"2024-04-19T02:03:35.102201+0000","last_peered":"2024-04-19T02:03:35.102201+0000","last_clean":"2024-04-19T02:03:35.102201+0000","last_became_active":"2024-04-19T02:03:11.673940+0000","last_became_peered":"2024-04-19T02:03:11.673940+0000","last_unstale":"2024-04-19T02:03:35.102201+0000","last_undegraded":"2024-04-19T02:03:35.102201+0000","last_fullsized":"2024-04-19T02:03:35.102201+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:18.862009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094453099999999997,"stat_sum":{"num_bytes":180224,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":530,"num_write":2,"num_write_kb":176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9,8,4,5,6,10],"acting":[0,1,9,8,4,5,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"},{"start":"d","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":66,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.050625+0000","last_change":"2024-04-19T02:03:02.570839+0000","last_active":"2024-04-19T02:03:34.050625+0000","last_peered":"2024-04-19T02:03:34.050625+0000","last_clean":"2024-04-19T02:03:34.050625+0000","last_became_active":"2024-04-19T02:03:02.570116+0000","last_became_peered":"2024-04-19T02:03:02.570116+0000","last_unstale":"2024-04-19T02:03:34.050625+0000","last_undegraded":"2024-04-19T02:03:34.050625+0000","last_fullsized":"2024-04-19T02:03:34.050625+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:23.134425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":63848448,"num_objects":50,"num_object_clones":6,"num_object_copies":400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":2,"num_read":281,"num_read_kb":90583,"num_write":444,"num_write_kb":79968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":125747200,"data_stored":127287296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":539,"ondisk_log_size":539,"up":128,"acting":128,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":19,"seq":81604378635,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38908,"kb_used_data":7720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332932,"statfs":{"total":96636764160,"available":96596922368,"internally_reserved":0,"allocated":7905280,"data_stored":7665009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":19,"seq":81604378634,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40276,"kb_used_data":9088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331564,"statfs":{"total":96636764160,"available":96595521536,"internally_reserved":0,"allocated":9306112,"data_stored":9258353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411338,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45416,"kb_used_data":14236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326424,"statfs":{"total":96636764160,"available":96590258176,"internally_reserved":0,"allocated":14577664,"data_stored":14337393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411339,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44040,"kb_used_data":12860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327800,"statfs":{"total":96636764160,"available":96591667200,"internally_reserved":0,"allocated":13168640,"data_stored":13120881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411339,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38512,"kb_used_data":11484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94333328,"statfs":{"total":96636764160,"available":96597327872,"internally_reserved":0,"allocated":11759616,"data_stored":11708305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411339,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39744,"kb_used_data":12720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94332096,"statfs":{"total":96636764160,"available":96596066304,"internally_reserved":0,"allocated":13025280,"data_stored":12981400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":18,"seq":77309411337,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31484,"kb_used_data":4460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340356,"statfs":{"total":96636764160,"available":96604524544,"internally_reserved":0,"allocated":4567040,"data_stored":4523160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411338,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37644,"kb_used_data":10620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334196,"statfs":{"total":96636764160,"available":96598216704,"internally_reserved":0,"allocated":10874880,"data_stored":10831000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":18,"seq":77309411338,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38096,"kb_used_data":11072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94333744,"statfs":{"total":96636764160,"available":96597753856,"internally_reserved":0,"allocated":11337728,"data_stored":11293848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":17,"seq":73014444042,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35616,"kb_used_data":8596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336224,"statfs":{"total":96636764160,"available":96600293376,"internally_reserved":0,"allocated":8802304,"data_stored":8762303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":17,"seq":73014444042,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44504,"kb_used_data":13324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327336,"statfs":{"total":96636764160,"available":96591192064,"internally_reserved":0,"allocated":13643776,"data_stored":13403832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":17,"seq":73014444042,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42012,"kb_used_data":10824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329828,"statfs":{"total":96636764160,"available":96593743872,"internally_reserved":0,"allocated":11083776,"data_stored":10847384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8753152,"data_stored":8945664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10543104,"data_stored":10543104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7352320,"data_stored":7352320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11911168,"data_stored":12103680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11448320,"data_stored":11640832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12570624,"data_stored":12570624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4026368,"data_stored":4218880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":13180928,"data_stored":13373440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":10674176,"data_stored":10866688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":12615680,"data_stored":12808192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":8646656,"data_stored":8839168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":14024704,"data_stored":14024704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T02:03:38.401 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T02:03:38.405 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:38.406+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:38.433 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:38.433 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:38.446 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-19T02:03:38.531 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:38.534+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:38.602 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:38.602 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:38.606 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:38.610+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:38.615 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-19T02:03:38.685 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:03:38.686 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-19T02:03:38.702 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-04-19T02:03:38.215120+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64438816,"num_objects":52,"num_object_clones":6,"num_object_copies":404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":2,"num_read":407,"num_read_kb":90692,"num_write":625,"num_write_kb":82906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":676,"ondisk_log_size":676,"up":146,"acting":146,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1132462080,"kb_used":476252,"kb_used_data":127004,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1131985828,"statfs":{"total":1159641169920,"available":1159153487872,"internally_reserved":0,"allocated":130052096,"data_stored":128732868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12550144,"num_objects":-5,"num_object_clones":-6,"num_object_copies":-40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5,"num_whiteouts":1,"num_read":46,"num_read_kb":44541,"num_write":82,"num_write_kb":18736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001317"},"pg_stats":[{"pgid":"3.f","version":"47'35","reported_seq":165,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.067421+0000","last_change":"2024-04-19T02:03:31.110932+0000","last_active":"2024-04-19T02:03:34.067421+0000","last_peered":"2024-04-19T02:03:34.067421+0000","last_clean":"2024-04-19T02:03:34.067421+0000","last_became_active":"2024-04-19T02:03:11.728136+0000","last_became_peered":"2024-04-19T02:03:11.728136+0000","last_unstale":"2024-04-19T02:03:34.067421+0000","last_undegraded":"2024-04-19T02:03:34.067421+0000","last_fullsized":"2024-04-19T02:03:34.067421+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:10.999100+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.02342029,"stat_sum":{"num_bytes":4816896,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":10857,"num_write":22,"num_write_kb":5712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,2,3,5,8,0,7],"acting":[11,4,2,3,5,8,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"3.e","version":"50'34","reported_seq":162,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.062028+0000","last_change":"2024-04-19T02:03:31.115711+0000","last_active":"2024-04-19T02:03:34.062028+0000","last_peered":"2024-04-19T02:03:34.062028+0000","last_clean":"2024-04-19T02:03:34.062028+0000","last_became_active":"2024-04-19T02:03:11.730135+0000","last_became_peered":"2024-04-19T02:03:11.730135+0000","last_unstale":"2024-04-19T02:03:34.062028+0000","last_undegraded":"2024-04-19T02:03:34.062028+0000","last_fullsized":"2024-04-19T02:03:34.062028+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:33.583038+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.017062461000000001,"stat_sum":{"num_bytes":2392064,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":5477,"num_write":15,"num_write_kb":3552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5,11,0,2,3,9],"acting":[7,4,5,11,0,2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"3.d","version":"50'49","reported_seq":183,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.104358+0000","last_change":"2024-04-19T02:03:31.100201+0000","last_active":"2024-04-19T02:03:34.104358+0000","last_peered":"2024-04-19T02:03:34.104358+0000","last_clean":"2024-04-19T02:03:34.104358+0000","last_became_active":"2024-04-19T02:03:11.632682+0000","last_became_peered":"2024-04-19T02:03:11.632682+0000","last_unstale":"2024-04-19T02:03:34.104358+0000","last_undegraded":"2024-04-19T02:03:34.104358+0000","last_fullsized":"2024-04-19T02:03:34.104358+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:16.173000+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.018754509999999999,"stat_sum":{"num_bytes":7880704,"num_objects":6,"num_object_clones":2,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":26,"num_read_kb":5818,"num_write":46,"num_write_kb":10128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7,4,5,3,9,0],"acting":[8,11,7,4,5,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"3.c","version":"50'28","reported_seq":165,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.056568+0000","last_change":"2024-04-19T02:03:31.094093+0000","last_active":"2024-04-19T02:03:34.056568+0000","last_peered":"2024-04-19T02:03:34.056568+0000","last_clean":"2024-04-19T02:03:34.056568+0000","last_became_active":"2024-04-19T02:03:11.731568+0000","last_became_peered":"2024-04-19T02:03:11.731568+0000","last_unstale":"2024-04-19T02:03:34.056568+0000","last_undegraded":"2024-04-19T02:03:34.056568+0000","last_fullsized":"2024-04-19T02:03:34.056568+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:33.581741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029319580000000001,"stat_sum":{"num_bytes":2473984,"num_objects":5,"num_object_clones":1,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":31,"num_read_kb":11050,"num_write":15,"num_write_kb":3920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,11,7,4,2,0,5],"acting":[9,3,11,7,4,2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"3.b","version":"47'38","reported_seq":166,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.058008+0000","last_change":"2024-04-19T02:03:31.121913+0000","last_active":"2024-04-19T02:03:34.058008+0000","last_peered":"2024-04-19T02:03:34.058008+0000","last_clean":"2024-04-19T02:03:34.058008+0000","last_became_active":"2024-04-19T02:03:11.677316+0000","last_became_peered":"2024-04-19T02:03:11.677316+0000","last_unstale":"2024-04-19T02:03:34.058008+0000","last_undegraded":"2024-04-19T02:03:34.058008+0000","last_fullsized":"2024-04-19T02:03:34.058008+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:30.430288+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.043083218,"stat_sum":{"num_bytes":2834432,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":22,"num_read_kb":6047,"num_write":17,"num_write_kb":5072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8,6,4,10,1,7],"acting":[3,0,8,6,4,10,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"3.a","version":"47'55","reported_seq":184,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.066597+0000","last_change":"2024-04-19T02:03:31.115131+0000","last_active":"2024-04-19T02:03:34.066597+0000","last_peered":"2024-04-19T02:03:34.066597+0000","last_clean":"2024-04-19T02:03:34.066597+0000","last_became_active":"2024-04-19T02:03:11.727996+0000","last_became_peered":"2024-04-19T02:03:11.727996+0000","last_unstale":"2024-04-19T02:03:34.066597+0000","last_undegraded":"2024-04-19T02:03:34.066597+0000","last_fullsized":"2024-04-19T02:03:34.066597+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:13.595764+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.030207358,"stat_sum":{"num_bytes":7864320,"num_objects":4,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":16131,"num_write":68,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,1,9,5,7,3,2],"acting":[11,8,1,9,5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"3.9","version":"47'23","reported_seq":148,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.047784+0000","last_change":"2024-04-19T02:03:31.100671+0000","last_active":"2024-04-19T02:03:34.047784+0000","last_peered":"2024-04-19T02:03:34.047784+0000","last_clean":"2024-04-19T02:03:34.047784+0000","last_became_active":"2024-04-19T02:03:11.729255+0000","last_became_peered":"2024-04-19T02:03:11.729255+0000","last_unstale":"2024-04-19T02:03:34.047784+0000","last_undegraded":"2024-04-19T02:03:34.047784+0000","last_fullsized":"2024-04-19T02:03:34.047784+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:14.014641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056021719999999999,"stat_sum":{"num_bytes":3833856,"num_objects":3,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":17,"num_write_kb":4480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,10,2,9,1,7,11],"acting":[4,3,10,2,9,1,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"3.8","version":"49'46","reported_seq":176,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.047628+0000","last_change":"2024-04-19T02:03:31.120264+0000","last_active":"2024-04-19T02:03:34.047628+0000","last_peered":"2024-04-19T02:03:34.047628+0000","last_clean":"2024-04-19T02:03:34.047628+0000","last_became_active":"2024-04-19T02:03:11.679384+0000","last_became_peered":"2024-04-19T02:03:11.679384+0000","last_unstale":"2024-04-19T02:03:34.047628+0000","last_undegraded":"2024-04-19T02:03:34.047628+0000","last_fullsized":"2024-04-19T02:03:34.047628+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:39.588360+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.032089586000000003,"stat_sum":{"num_bytes":6193152,"num_objects":4,"num_object_clones":1,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":10138,"num_write":23,"num_write_kb":6192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7,11,5,9,6,10],"acting":[4,1,7,11,5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"1.0","version":"21'137","reported_seq":245,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.049982+0000","last_change":"2024-04-19T02:02:59.974166+0000","last_active":"2024-04-19T02:03:34.049982+0000","last_peered":"2024-04-19T02:03:34.049982+0000","last_clean":"2024-04-19T02:03:34.049982+0000","last_became_active":"2024-04-19T02:02:59.972005+0000","last_became_peered":"2024-04-19T02:02:59.972005+0000","last_unstale":"2024-04-19T02:03:34.049982+0000","last_undegraded":"2024-04-19T02:03:34.049982+0000","last_fullsized":"2024-04-19T02:03:34.049982+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:02:58.486225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:02:58.486225+0000","last_clean_scrub_stamp":"2024-04-19T02:02:58.486225+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:20.275707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":66,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.066268+0000","last_change":"2024-04-19T02:03:03.026806+0000","last_active":"2024-04-19T02:03:34.066268+0000","last_peered":"2024-04-19T02:03:34.066268+0000","last_clean":"2024-04-19T02:03:34.066268+0000","last_became_active":"2024-04-19T02:03:03.026063+0000","last_became_peered":"2024-04-19T02:03:03.026063+0000","last_unstale":"2024-04-19T02:03:34.066268+0000","last_undegraded":"2024-04-19T02:03:34.066268+0000","last_fullsized":"2024-04-19T02:03:34.066268+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:19.673016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"50'98","reported_seq":239,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.091760+0000","last_change":"2024-04-19T02:03:31.132520+0000","last_active":"2024-04-19T02:03:34.091760+0000","last_peered":"2024-04-19T02:03:34.091760+0000","last_clean":"2024-04-19T02:03:34.091760+0000","last_became_active":"2024-04-19T02:03:11.630475+0000","last_became_peered":"2024-04-19T02:03:11.630475+0000","last_unstale":"2024-04-19T02:03:34.091760+0000","last_undegraded":"2024-04-19T02:03:34.091760+0000","last_fullsized":"2024-04-19T02:03:34.091760+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:12.351889+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.053768872000000002,"stat_sum":{"num_bytes":8650752,"num_objects":8,"num_object_clones":1,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":49,"num_read_kb":14362,"num_write":87,"num_write_kb":12640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10,4,8,1,11,9],"acting":[3,5,10,4,8,1,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"2.0","version":"0'0","reported_seq":66,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.049970+0000","last_change":"2024-04-19T02:03:02.561076+0000","last_active":"2024-04-19T02:03:34.049970+0000","last_peered":"2024-04-19T02:03:34.049970+0000","last_clean":"2024-04-19T02:03:34.049970+0000","last_became_active":"2024-04-19T02:03:02.560210+0000","last_became_peered":"2024-04-19T02:03:02.560210+0000","last_unstale":"2024-04-19T02:03:34.049970+0000","last_undegraded":"2024-04-19T02:03:34.049970+0000","last_fullsized":"2024-04-19T02:03:34.049970+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:16.900004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"47'21","reported_seq":155,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-19T02:03:35.101457+0000","last_change":"2024-04-19T02:03:35.102074+0000","last_active":"2024-04-19T02:03:35.101457+0000","last_peered":"2024-04-19T02:03:35.101457+0000","last_clean":"2024-04-19T02:03:35.101457+0000","last_became_active":"2024-04-19T02:03:11.633394+0000","last_became_peered":"2024-04-19T02:03:11.633394+0000","last_unstale":"2024-04-19T02:03:35.101457+0000","last_undegraded":"2024-04-19T02:03:35.101457+0000","last_fullsized":"2024-04-19T02:03:35.101457+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:30.509710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085909099999999998,"stat_sum":{"num_bytes":4096000,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":1350,"num_write":15,"num_write_kb":4528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,4,5,10,7,9,1],"acting":[0,11,4,5,10,7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"},{"start":"d","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":61,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.057488+0000","last_change":"2024-04-19T02:03:09.594947+0000","last_active":"2024-04-19T02:03:34.057488+0000","last_peered":"2024-04-19T02:03:34.057488+0000","last_clean":"2024-04-19T02:03:34.057488+0000","last_became_active":"2024-04-19T02:03:09.594072+0000","last_became_peered":"2024-04-19T02:03:09.594072+0000","last_unstale":"2024-04-19T02:03:34.057488+0000","last_undegraded":"2024-04-19T02:03:34.057488+0000","last_fullsized":"2024-04-19T02:03:34.057488+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:29.461650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":117,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T02:03:33.053493+0000","last_change":"2024-04-19T02:03:31.096841+0000","last_active":"2024-04-19T02:03:33.053493+0000","last_peered":"2024-04-19T02:03:33.053493+0000","last_clean":"2024-04-19T02:03:33.053493+0000","last_became_active":"2024-04-19T02:03:11.678078+0000","last_became_peered":"2024-04-19T02:03:11.678078+0000","last_unstale":"2024-04-19T02:03:33.053493+0000","last_undegraded":"2024-04-19T02:03:33.053493+0000","last_fullsized":"2024-04-19T02:03:33.053493+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:25.778027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039275129999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,9,1,6,4,0,10],"acting":[5,7,9,1,6,4,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"2.2","version":"0'0","reported_seq":66,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.050250+0000","last_change":"2024-04-19T02:03:02.560133+0000","last_active":"2024-04-19T02:03:34.050250+0000","last_peered":"2024-04-19T02:03:34.050250+0000","last_clean":"2024-04-19T02:03:34.050250+0000","last_became_active":"2024-04-19T02:03:02.559377+0000","last_became_peered":"2024-04-19T02:03:02.559377+0000","last_unstale":"2024-04-19T02:03:34.050250+0000","last_undegraded":"2024-04-19T02:03:34.050250+0000","last_fullsized":"2024-04-19T02:03:34.050250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:30.728400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"48'15","reported_seq":136,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T02:03:33.063555+0000","last_change":"2024-04-19T02:03:31.119565+0000","last_active":"2024-04-19T02:03:33.063555+0000","last_peered":"2024-04-19T02:03:33.063555+0000","last_clean":"2024-04-19T02:03:33.063555+0000","last_became_active":"2024-04-19T02:03:11.677099+0000","last_became_peered":"2024-04-19T02:03:11.677099+0000","last_unstale":"2024-04-19T02:03:33.063555+0000","last_undegraded":"2024-04-19T02:03:33.063555+0000","last_fullsized":"2024-04-19T02:03:33.063555+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:36.180333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026550929999999999,"stat_sum":{"num_bytes":2441216,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":2368,"num_write":49,"num_write_kb":2384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,0,3,4,7,9,6],"acting":[10,8,0,3,4,7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"3.5","version":"45'33","reported_seq":151,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T02:03:33.063294+0000","last_change":"2024-04-19T02:03:31.116943+0000","last_active":"2024-04-19T02:03:33.063294+0000","last_peered":"2024-04-19T02:03:33.063294+0000","last_clean":"2024-04-19T02:03:33.063294+0000","last_became_active":"2024-04-19T02:03:11.728949+0000","last_became_peered":"2024-04-19T02:03:11.728949+0000","last_unstale":"2024-04-19T02:03:33.063294+0000","last_undegraded":"2024-04-19T02:03:33.063294+0000","last_fullsized":"2024-04-19T02:03:33.063294+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:28.041143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.02327926,"stat_sum":{"num_bytes":2293760,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":36,"num_write_kb":3584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2,9,8,1,6,5],"acting":[10,3,2,9,8,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"2.4","version":"0'0","reported_seq":64,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T02:03:33.053292+0000","last_change":"2024-04-19T02:03:02.597582+0000","last_active":"2024-04-19T02:03:33.053292+0000","last_peered":"2024-04-19T02:03:33.053292+0000","last_clean":"2024-04-19T02:03:33.053292+0000","last_became_active":"2024-04-19T02:03:02.597118+0000","last_became_peered":"2024-04-19T02:03:02.597118+0000","last_unstale":"2024-04-19T02:03:33.053292+0000","last_undegraded":"2024-04-19T02:03:33.053292+0000","last_fullsized":"2024-04-19T02:03:33.053292+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:13.683143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"47'48","reported_seq":174,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T02:03:33.053383+0000","last_change":"2024-04-19T02:03:31.086845+0000","last_active":"2024-04-19T02:03:33.053383+0000","last_peered":"2024-04-19T02:03:33.053383+0000","last_clean":"2024-04-19T02:03:33.053383+0000","last_became_active":"2024-04-19T02:03:11.730165+0000","last_became_peered":"2024-04-19T02:03:11.730165+0000","last_unstale":"2024-04-19T02:03:33.053383+0000","last_undegraded":"2024-04-19T02:03:33.053383+0000","last_fullsized":"2024-04-19T02:03:33.053383+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:14.215941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010297920000000001,"stat_sum":{"num_bytes":5734400,"num_objects":5,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":23,"num_read_kb":6442,"num_write":26,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,10,8,7,2,11,0],"acting":[1,5,10,8,7,2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"2.5","version":"0'0","reported_seq":66,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.062019+0000","last_change":"2024-04-19T02:03:02.595929+0000","last_active":"2024-04-19T02:03:34.062019+0000","last_peered":"2024-04-19T02:03:34.062019+0000","last_clean":"2024-04-19T02:03:34.062019+0000","last_became_active":"2024-04-19T02:03:02.595232+0000","last_became_peered":"2024-04-19T02:03:02.595232+0000","last_unstale":"2024-04-19T02:03:34.062019+0000","last_undegraded":"2024-04-19T02:03:34.062019+0000","last_fullsized":"2024-04-19T02:03:34.062019+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:04.077370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"46'14","reported_seq":135,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.058776+0000","last_change":"2024-04-19T02:03:31.122442+0000","last_active":"2024-04-19T02:03:34.058776+0000","last_peered":"2024-04-19T02:03:34.058776+0000","last_clean":"2024-04-19T02:03:34.058776+0000","last_became_active":"2024-04-19T02:03:11.675056+0000","last_became_peered":"2024-04-19T02:03:11.675056+0000","last_unstale":"2024-04-19T02:03:34.058776+0000","last_undegraded":"2024-04-19T02:03:34.058776+0000","last_fullsized":"2024-04-19T02:03:34.058776+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:12.521007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.043351122999999998,"stat_sum":{"num_bytes":2162688,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":6,"num_write_kb":2496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,9,0,7,5,11,1],"acting":[3,6,9,0,7,5,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"2.6","version":"0'0","reported_seq":64,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T02:03:33.053459+0000","last_change":"2024-04-19T02:03:02.988302+0000","last_active":"2024-04-19T02:03:33.053459+0000","last_peered":"2024-04-19T02:03:33.053459+0000","last_clean":"2024-04-19T02:03:33.053459+0000","last_became_active":"2024-04-19T02:03:02.988073+0000","last_became_peered":"2024-04-19T02:03:02.988073+0000","last_unstale":"2024-04-19T02:03:33.053459+0000","last_undegraded":"2024-04-19T02:03:33.053459+0000","last_fullsized":"2024-04-19T02:03:33.053459+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:28.455271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"29'2","reported_seq":133,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-19T02:03:35.102201+0000","last_change":"2024-04-19T02:03:35.102794+0000","last_active":"2024-04-19T02:03:35.102201+0000","last_peered":"2024-04-19T02:03:35.102201+0000","last_clean":"2024-04-19T02:03:35.102201+0000","last_became_active":"2024-04-19T02:03:11.673940+0000","last_became_peered":"2024-04-19T02:03:11.673940+0000","last_unstale":"2024-04-19T02:03:35.102201+0000","last_undegraded":"2024-04-19T02:03:35.102201+0000","last_fullsized":"2024-04-19T02:03:35.102201+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:18.862009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094453099999999997,"stat_sum":{"num_bytes":180224,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":530,"num_write":2,"num_write_kb":176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9,8,4,5,6,10],"acting":[0,1,9,8,4,5,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"},{"start":"d","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":66,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.050625+0000","last_change":"2024-04-19T02:03:02.570839+0000","last_active":"2024-04-19T02:03:34.050625+0000","last_peered":"2024-04-19T02:03:34.050625+0000","last_clean":"2024-04-19T02:03:34.050625+0000","last_became_active":"2024-04-19T02:03:02.570116+0000","last_became_peered":"2024-04-19T02:03:02.570116+0000","last_unstale":"2024-04-19T02:03:34.050625+0000","last_undegraded":"2024-04-19T02:03:34.050625+0000","last_fullsized":"2024-04-19T02:03:34.050625+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:23.134425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":63848448,"num_objects":50,"num_object_clones":6,"num_object_copies":400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":2,"num_read":281,"num_read_kb":90583,"num_write":444,"num_write_kb":79968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":125747200,"data_stored":127287296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":539,"ondisk_log_size":539,"up":128,"acting":128,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":19,"seq":81604378635,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38908,"kb_used_data":7720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332932,"statfs":{"total":96636764160,"available":96596922368,"internally_reserved":0,"allocated":7905280,"data_stored":7665009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":19,"seq":81604378634,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40276,"kb_used_data":9088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331564,"statfs":{"total":96636764160,"available":96595521536,"internally_reserved":0,"allocated":9306112,"data_stored":9258353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411338,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45416,"kb_used_data":14236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326424,"statfs":{"total":96636764160,"available":96590258176,"internally_reserved":0,"allocated":14577664,"data_stored":14337393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411339,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44040,"kb_used_data":12860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327800,"statfs":{"total":96636764160,"available":96591667200,"internally_reserved":0,"allocated":13168640,"data_stored":13120881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411339,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38512,"kb_used_data":11484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94333328,"statfs":{"total":96636764160,"available":96597327872,"internally_reserved":0,"allocated":11759616,"data_stored":11708305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411339,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39744,"kb_used_data":12720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94332096,"statfs":{"total":96636764160,"available":96596066304,"internally_reserved":0,"allocated":13025280,"data_stored":12981400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":18,"seq":77309411337,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31484,"kb_used_data":4460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340356,"statfs":{"total":96636764160,"available":96604524544,"internally_reserved":0,"allocated":4567040,"data_stored":4523160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411338,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37644,"kb_used_data":10620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334196,"statfs":{"total":96636764160,"available":96598216704,"internally_reserved":0,"allocated":10874880,"data_stored":10831000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":18,"seq":77309411338,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38096,"kb_used_data":11072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94333744,"statfs":{"total":96636764160,"available":96597753856,"internally_reserved":0,"allocated":11337728,"data_stored":11293848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":17,"seq":73014444042,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35616,"kb_used_data":8596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336224,"statfs":{"total":96636764160,"available":96600293376,"internally_reserved":0,"allocated":8802304,"data_stored":8762303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":17,"seq":73014444042,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44504,"kb_used_data":13324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327336,"statfs":{"total":96636764160,"available":96591192064,"internally_reserved":0,"allocated":13643776,"data_stored":13403832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":17,"seq":73014444042,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42012,"kb_used_data":10824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329828,"statfs":{"total":96636764160,"available":96593743872,"internally_reserved":0,"allocated":11083776,"data_stored":10847384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8753152,"data_stored":8945664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10543104,"data_stored":10543104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7352320,"data_stored":7352320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11911168,"data_stored":12103680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11448320,"data_stored":11640832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12570624,"data_stored":12570624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4026368,"data_stored":4218880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":13180928,"data_stored":13373440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":10674176,"data_stored":10866688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":12615680,"data_stored":12808192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":8646656,"data_stored":8839168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":14024704,"data_stored":14024704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T02:03:38.704 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-19T02:03:38.704 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-19T02:03:38.704 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-04-19T02:03:38.705 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T02:03:38.707 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:38.710+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:38.785 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:38.785 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:38.799 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-19T02:03:38.808 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:38.810+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:38.902 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T02:03:38.909 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:38.910+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:38.954 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:38.954 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:38.969 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-19T02:03:39.010 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:39.006+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:39.031 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-04-19T02:03:39.031 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T02:03:39.032 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:03:39.033 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-19T02:03:39.043 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is unset 2024-04-19T02:03:39.048 INFO:tasks.rados.rados.0.smithi059.stdout:216: done (2 left) 2024-04-19T02:03:39.048 INFO:tasks.rados.rados.0.smithi059.stdout:217: read oid 27 snap -1 2024-04-19T02:03:39.048 INFO:tasks.rados.rados.0.smithi059.stdout:217: expect deleted 2024-04-19T02:03:39.048 INFO:tasks.rados.rados.0.smithi059.stdout:218: setattr oid 44 current snap is 14 2024-04-19T02:03:39.050 INFO:tasks.rados.rados.0.smithi059.stdout:219: rmattr oid 11 current snap is 14 2024-04-19T02:03:39.050 INFO:tasks.rados.rados.0.smithi059.stdout:219: done (4 left) 2024-04-19T02:03:39.050 INFO:tasks.rados.rados.0.smithi059.stdout:220: read oid 42 snap -1 2024-04-19T02:03:39.051 INFO:tasks.rados.rados.0.smithi059.stdout:220: expect (ObjNum 56 snap 1 seq_num 56) 2024-04-19T02:03:39.051 INFO:tasks.rados.rados.0.smithi059.stdout:221: read oid 30 snap 10 2024-04-19T02:03:39.051 INFO:tasks.rados.rados.0.smithi059.stdout:221: expect (ObjNum 31 snap 0 seq_num 31) 2024-04-19T02:03:39.051 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 41 current snap is 14 2024-04-19T02:03:39.051 INFO:tasks.rados.rados.0.smithi059.stdout:222: seq_num 105 ranges {3293184=606208,3899392=688128,4587520=475136} 2024-04-19T02:03:39.052 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-04-19T02:03:39.052 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T02:03:39.055 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-04-19T02:03:38.215120+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64438816,"num_objects":52,"num_object_clones":6,"num_object_copies":404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":2,"num_read":407,"num_read_kb":90692,"num_write":625,"num_write_kb":82906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":676,"ondisk_log_size":676,"up":146,"acting":146,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1132462080,"kb_used":476252,"kb_used_data":127004,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1131985828,"statfs":{"total":1159641169920,"available":1159153487872,"internally_reserved":0,"allocated":130052096,"data_stored":128732868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12550144,"num_objects":-5,"num_object_clones":-6,"num_object_copies":-40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5,"num_whiteouts":1,"num_read":46,"num_read_kb":44541,"num_write":82,"num_write_kb":18736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001317"},"pg_stats":[{"pgid":"3.f","version":"47'35","reported_seq":165,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.067421+0000","last_change":"2024-04-19T02:03:31.110932+0000","last_active":"2024-04-19T02:03:34.067421+0000","last_peered":"2024-04-19T02:03:34.067421+0000","last_clean":"2024-04-19T02:03:34.067421+0000","last_became_active":"2024-04-19T02:03:11.728136+0000","last_became_peered":"2024-04-19T02:03:11.728136+0000","last_unstale":"2024-04-19T02:03:34.067421+0000","last_undegraded":"2024-04-19T02:03:34.067421+0000","last_fullsized":"2024-04-19T02:03:34.067421+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:10.999100+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.02342029,"stat_sum":{"num_bytes":4816896,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":10857,"num_write":22,"num_write_kb":5712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,2,3,5,8,0,7],"acting":[11,4,2,3,5,8,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"3.e","version":"50'34","reported_seq":162,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.062028+0000","last_change":"2024-04-19T02:03:31.115711+0000","last_active":"2024-04-19T02:03:34.062028+0000","last_peered":"2024-04-19T02:03:34.062028+0000","last_clean":"2024-04-19T02:03:34.062028+0000","last_became_active":"2024-04-19T02:03:11.730135+0000","last_became_peered":"2024-04-19T02:03:11.730135+0000","last_unstale":"2024-04-19T02:03:34.062028+0000","last_undegraded":"2024-04-19T02:03:34.062028+0000","last_fullsized":"2024-04-19T02:03:34.062028+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:33.583038+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.017062461000000001,"stat_sum":{"num_bytes":2392064,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":5477,"num_write":15,"num_write_kb":3552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5,11,0,2,3,9],"acting":[7,4,5,11,0,2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"3.d","version":"50'49","reported_seq":183,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.104358+0000","last_change":"2024-04-19T02:03:31.100201+0000","last_active":"2024-04-19T02:03:34.104358+0000","last_peered":"2024-04-19T02:03:34.104358+0000","last_clean":"2024-04-19T02:03:34.104358+0000","last_became_active":"2024-04-19T02:03:11.632682+0000","last_became_peered":"2024-04-19T02:03:11.632682+0000","last_unstale":"2024-04-19T02:03:34.104358+0000","last_undegraded":"2024-04-19T02:03:34.104358+0000","last_fullsized":"2024-04-19T02:03:34.104358+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:16.173000+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.018754509999999999,"stat_sum":{"num_bytes":7880704,"num_objects":6,"num_object_clones":2,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":26,"num_read_kb":5818,"num_write":46,"num_write_kb":10128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7,4,5,3,9,0],"acting":[8,11,7,4,5,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"3.c","version":"50'28","reported_seq":165,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.056568+0000","last_change":"2024-04-19T02:03:31.094093+0000","last_active":"2024-04-19T02:03:34.056568+0000","last_peered":"2024-04-19T02:03:34.056568+0000","last_clean":"2024-04-19T02:03:34.056568+0000","last_became_active":"2024-04-19T02:03:11.731568+0000","last_became_peered":"2024-04-19T02:03:11.731568+0000","last_unstale":"2024-04-19T02:03:34.056568+0000","last_undegraded":"2024-04-19T02:03:34.056568+0000","last_fullsized":"2024-04-19T02:03:34.056568+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:33.581741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029319580000000001,"stat_sum":{"num_bytes":2473984,"num_objects":5,"num_object_clones":1,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":31,"num_read_kb":11050,"num_write":15,"num_write_kb":3920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,11,7,4,2,0,5],"acting":[9,3,11,7,4,2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"3.b","version":"47'38","reported_seq":166,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.058008+0000","last_change":"2024-04-19T02:03:31.121913+0000","last_active":"2024-04-19T02:03:34.058008+0000","last_peered":"2024-04-19T02:03:34.058008+0000","last_clean":"2024-04-19T02:03:34.058008+0000","last_became_active":"2024-04-19T02:03:11.677316+0000","last_became_peered":"2024-04-19T02:03:11.677316+0000","last_unstale":"2024-04-19T02:03:34.058008+0000","last_undegraded":"2024-04-19T02:03:34.058008+0000","last_fullsized":"2024-04-19T02:03:34.058008+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:30.430288+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.043083218,"stat_sum":{"num_bytes":2834432,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":22,"num_read_kb":6047,"num_write":17,"num_write_kb":5072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8,6,4,10,1,7],"acting":[3,0,8,6,4,10,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"3.a","version":"47'55","reported_seq":184,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.066597+0000","last_change":"2024-04-19T02:03:31.115131+0000","last_active":"2024-04-19T02:03:34.066597+0000","last_peered":"2024-04-19T02:03:34.066597+0000","last_clean":"2024-04-19T02:03:34.066597+0000","last_became_active":"2024-04-19T02:03:11.727996+0000","last_became_peered":"2024-04-19T02:03:11.727996+0000","last_unstale":"2024-04-19T02:03:34.066597+0000","last_undegraded":"2024-04-19T02:03:34.066597+0000","last_fullsized":"2024-04-19T02:03:34.066597+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:13.595764+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.030207358,"stat_sum":{"num_bytes":7864320,"num_objects":4,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":16131,"num_write":68,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,1,9,5,7,3,2],"acting":[11,8,1,9,5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"3.9","version":"47'23","reported_seq":148,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.047784+0000","last_change":"2024-04-19T02:03:31.100671+0000","last_active":"2024-04-19T02:03:34.047784+0000","last_peered":"2024-04-19T02:03:34.047784+0000","last_clean":"2024-04-19T02:03:34.047784+0000","last_became_active":"2024-04-19T02:03:11.729255+0000","last_became_peered":"2024-04-19T02:03:11.729255+0000","last_unstale":"2024-04-19T02:03:34.047784+0000","last_undegraded":"2024-04-19T02:03:34.047784+0000","last_fullsized":"2024-04-19T02:03:34.047784+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:14.014641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056021719999999999,"stat_sum":{"num_bytes":3833856,"num_objects":3,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":17,"num_write_kb":4480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,10,2,9,1,7,11],"acting":[4,3,10,2,9,1,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"3.8","version":"49'46","reported_seq":176,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.047628+0000","last_change":"2024-04-19T02:03:31.120264+0000","last_active":"2024-04-19T02:03:34.047628+0000","last_peered":"2024-04-19T02:03:34.047628+0000","last_clean":"2024-04-19T02:03:34.047628+0000","last_became_active":"2024-04-19T02:03:11.679384+0000","last_became_peered":"2024-04-19T02:03:11.679384+0000","last_unstale":"2024-04-19T02:03:34.047628+0000","last_undegraded":"2024-04-19T02:03:34.047628+0000","last_fullsized":"2024-04-19T02:03:34.047628+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:39.588360+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.032089586000000003,"stat_sum":{"num_bytes":6193152,"num_objects":4,"num_object_clones":1,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":10138,"num_write":23,"num_write_kb":6192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7,11,5,9,6,10],"acting":[4,1,7,11,5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"1.0","version":"21'137","reported_seq":245,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.049982+0000","last_change":"2024-04-19T02:02:59.974166+0000","last_active":"2024-04-19T02:03:34.049982+0000","last_peered":"2024-04-19T02:03:34.049982+0000","last_clean":"2024-04-19T02:03:34.049982+0000","last_became_active":"2024-04-19T02:02:59.972005+0000","last_became_peered":"2024-04-19T02:02:59.972005+0000","last_unstale":"2024-04-19T02:03:34.049982+0000","last_undegraded":"2024-04-19T02:03:34.049982+0000","last_fullsized":"2024-04-19T02:03:34.049982+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:02:58.486225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:02:58.486225+0000","last_clean_scrub_stamp":"2024-04-19T02:02:58.486225+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:20.275707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":66,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.066268+0000","last_change":"2024-04-19T02:03:03.026806+0000","last_active":"2024-04-19T02:03:34.066268+0000","last_peered":"2024-04-19T02:03:34.066268+0000","last_clean":"2024-04-19T02:03:34.066268+0000","last_became_active":"2024-04-19T02:03:03.026063+0000","last_became_peered":"2024-04-19T02:03:03.026063+0000","last_unstale":"2024-04-19T02:03:34.066268+0000","last_undegraded":"2024-04-19T02:03:34.066268+0000","last_fullsized":"2024-04-19T02:03:34.066268+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:19.673016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"50'98","reported_seq":239,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.091760+0000","last_change":"2024-04-19T02:03:31.132520+0000","last_active":"2024-04-19T02:03:34.091760+0000","last_peered":"2024-04-19T02:03:34.091760+0000","last_clean":"2024-04-19T02:03:34.091760+0000","last_became_active":"2024-04-19T02:03:11.630475+0000","last_became_peered":"2024-04-19T02:03:11.630475+0000","last_unstale":"2024-04-19T02:03:34.091760+0000","last_undegraded":"2024-04-19T02:03:34.091760+0000","last_fullsized":"2024-04-19T02:03:34.091760+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:12.351889+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.053768872000000002,"stat_sum":{"num_bytes":8650752,"num_objects":8,"num_object_clones":1,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":49,"num_read_kb":14362,"num_write":87,"num_write_kb":12640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10,4,8,1,11,9],"acting":[3,5,10,4,8,1,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"2.0","version":"0'0","reported_seq":66,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.049970+0000","last_change":"2024-04-19T02:03:02.561076+0000","last_active":"2024-04-19T02:03:34.049970+0000","last_peered":"2024-04-19T02:03:34.049970+0000","last_clean":"2024-04-19T02:03:34.049970+0000","last_became_active":"2024-04-19T02:03:02.560210+0000","last_became_peered":"2024-04-19T02:03:02.560210+0000","last_unstale":"2024-04-19T02:03:34.049970+0000","last_undegraded":"2024-04-19T02:03:34.049970+0000","last_fullsized":"2024-04-19T02:03:34.049970+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:16.900004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"47'21","reported_seq":155,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-19T02:03:35.101457+0000","last_change":"2024-04-19T02:03:35.102074+0000","last_active":"2024-04-19T02:03:35.101457+0000","last_peered":"2024-04-19T02:03:35.101457+0000","last_clean":"2024-04-19T02:03:35.101457+0000","last_became_active":"2024-04-19T02:03:11.633394+0000","last_became_peered":"2024-04-19T02:03:11.633394+0000","last_unstale":"2024-04-19T02:03:35.101457+0000","last_undegraded":"2024-04-19T02:03:35.101457+0000","last_fullsized":"2024-04-19T02:03:35.101457+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:30.509710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085909099999999998,"stat_sum":{"num_bytes":4096000,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":1350,"num_write":15,"num_write_kb":4528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,4,5,10,7,9,1],"acting":[0,11,4,5,10,7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"},{"start":"d","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":61,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.057488+0000","last_change":"2024-04-19T02:03:09.594947+0000","last_active":"2024-04-19T02:03:34.057488+0000","last_peered":"2024-04-19T02:03:34.057488+0000","last_clean":"2024-04-19T02:03:34.057488+0000","last_became_active":"2024-04-19T02:03:09.594072+0000","last_became_peered":"2024-04-19T02:03:09.594072+0000","last_unstale":"2024-04-19T02:03:34.057488+0000","last_undegraded":"2024-04-19T02:03:34.057488+0000","last_fullsized":"2024-04-19T02:03:34.057488+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:29.461650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":117,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T02:03:33.053493+0000","last_change":"2024-04-19T02:03:31.096841+0000","last_active":"2024-04-19T02:03:33.053493+0000","last_peered":"2024-04-19T02:03:33.053493+0000","last_clean":"2024-04-19T02:03:33.053493+0000","last_became_active":"2024-04-19T02:03:11.678078+0000","last_became_peered":"2024-04-19T02:03:11.678078+0000","last_unstale":"2024-04-19T02:03:33.053493+0000","last_undegraded":"2024-04-19T02:03:33.053493+0000","last_fullsized":"2024-04-19T02:03:33.053493+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:25.778027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039275129999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,9,1,6,4,0,10],"acting":[5,7,9,1,6,4,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"2.2","version":"0'0","reported_seq":66,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.050250+0000","last_change":"2024-04-19T02:03:02.560133+0000","last_active":"2024-04-19T02:03:34.050250+0000","last_peered":"2024-04-19T02:03:34.050250+0000","last_clean":"2024-04-19T02:03:34.050250+0000","last_became_active":"2024-04-19T02:03:02.559377+0000","last_became_peered":"2024-04-19T02:03:02.559377+0000","last_unstale":"2024-04-19T02:03:34.050250+0000","last_undegraded":"2024-04-19T02:03:34.050250+0000","last_fullsized":"2024-04-19T02:03:34.050250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:30.728400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"48'15","reported_seq":136,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T02:03:33.063555+0000","last_change":"2024-04-19T02:03:31.119565+0000","last_active":"2024-04-19T02:03:33.063555+0000","last_peered":"2024-04-19T02:03:33.063555+0000","last_clean":"2024-04-19T02:03:33.063555+0000","last_became_active":"2024-04-19T02:03:11.677099+0000","last_became_peered":"2024-04-19T02:03:11.677099+0000","last_unstale":"2024-04-19T02:03:33.063555+0000","last_undegraded":"2024-04-19T02:03:33.063555+0000","last_fullsized":"2024-04-19T02:03:33.063555+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:36.180333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026550929999999999,"stat_sum":{"num_bytes":2441216,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":2368,"num_write":49,"num_write_kb":2384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,0,3,4,7,9,6],"acting":[10,8,0,3,4,7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"3.5","version":"45'33","reported_seq":151,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T02:03:33.063294+0000","last_change":"2024-04-19T02:03:31.116943+0000","last_active":"2024-04-19T02:03:33.063294+0000","last_peered":"2024-04-19T02:03:33.063294+0000","last_clean":"2024-04-19T02:03:33.063294+0000","last_became_active":"2024-04-19T02:03:11.728949+0000","last_became_peered":"2024-04-19T02:03:11.728949+0000","last_unstale":"2024-04-19T02:03:33.063294+0000","last_undegraded":"2024-04-19T02:03:33.063294+0000","last_fullsized":"2024-04-19T02:03:33.063294+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:28.041143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.02327926,"stat_sum":{"num_bytes":2293760,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":36,"num_write_kb":3584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2,9,8,1,6,5],"acting":[10,3,2,9,8,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"2.4","version":"0'0","reported_seq":64,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T02:03:33.053292+0000","last_change":"2024-04-19T02:03:02.597582+0000","last_active":"2024-04-19T02:03:33.053292+0000","last_peered":"2024-04-19T02:03:33.053292+0000","last_clean":"2024-04-19T02:03:33.053292+0000","last_became_active":"2024-04-19T02:03:02.597118+0000","last_became_peered":"2024-04-19T02:03:02.597118+0000","last_unstale":"2024-04-19T02:03:33.053292+0000","last_undegraded":"2024-04-19T02:03:33.053292+0000","last_fullsized":"2024-04-19T02:03:33.053292+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:13.683143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"47'48","reported_seq":174,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T02:03:33.053383+0000","last_change":"2024-04-19T02:03:31.086845+0000","last_active":"2024-04-19T02:03:33.053383+0000","last_peered":"2024-04-19T02:03:33.053383+0000","last_clean":"2024-04-19T02:03:33.053383+0000","last_became_active":"2024-04-19T02:03:11.730165+0000","last_became_peered":"2024-04-19T02:03:11.730165+0000","last_unstale":"2024-04-19T02:03:33.053383+0000","last_undegraded":"2024-04-19T02:03:33.053383+0000","last_fullsized":"2024-04-19T02:03:33.053383+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:14.215941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010297920000000001,"stat_sum":{"num_bytes":5734400,"num_objects":5,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":23,"num_read_kb":6442,"num_write":26,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,10,8,7,2,11,0],"acting":[1,5,10,8,7,2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"2.5","version":"0'0","reported_seq":66,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.062019+0000","last_change":"2024-04-19T02:03:02.595929+0000","last_active":"2024-04-19T02:03:34.062019+0000","last_peered":"2024-04-19T02:03:34.062019+0000","last_clean":"2024-04-19T02:03:34.062019+0000","last_became_active":"2024-04-19T02:03:02.595232+0000","last_became_peered":"2024-04-19T02:03:02.595232+0000","last_unstale":"2024-04-19T02:03:34.062019+0000","last_undegraded":"2024-04-19T02:03:34.062019+0000","last_fullsized":"2024-04-19T02:03:34.062019+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:04.077370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"46'14","reported_seq":135,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.058776+0000","last_change":"2024-04-19T02:03:31.122442+0000","last_active":"2024-04-19T02:03:34.058776+0000","last_peered":"2024-04-19T02:03:34.058776+0000","last_clean":"2024-04-19T02:03:34.058776+0000","last_became_active":"2024-04-19T02:03:11.675056+0000","last_became_peered":"2024-04-19T02:03:11.675056+0000","last_unstale":"2024-04-19T02:03:34.058776+0000","last_undegraded":"2024-04-19T02:03:34.058776+0000","last_fullsized":"2024-04-19T02:03:34.058776+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:12.521007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.043351122999999998,"stat_sum":{"num_bytes":2162688,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":6,"num_write_kb":2496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,9,0,7,5,11,1],"acting":[3,6,9,0,7,5,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"}]},{"pgid":"2.6","version":"0'0","reported_seq":64,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-19T02:03:33.053459+0000","last_change":"2024-04-19T02:03:02.988302+0000","last_active":"2024-04-19T02:03:33.053459+0000","last_peered":"2024-04-19T02:03:33.053459+0000","last_clean":"2024-04-19T02:03:33.053459+0000","last_became_active":"2024-04-19T02:03:02.988073+0000","last_became_peered":"2024-04-19T02:03:02.988073+0000","last_unstale":"2024-04-19T02:03:33.053459+0000","last_undegraded":"2024-04-19T02:03:33.053459+0000","last_fullsized":"2024-04-19T02:03:33.053459+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:28.455271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"29'2","reported_seq":133,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-19T02:03:35.102201+0000","last_change":"2024-04-19T02:03:35.102794+0000","last_active":"2024-04-19T02:03:35.102201+0000","last_peered":"2024-04-19T02:03:35.102201+0000","last_clean":"2024-04-19T02:03:35.102201+0000","last_became_active":"2024-04-19T02:03:11.673940+0000","last_became_peered":"2024-04-19T02:03:11.673940+0000","last_unstale":"2024-04-19T02:03:35.102201+0000","last_undegraded":"2024-04-19T02:03:35.102201+0000","last_fullsized":"2024-04-19T02:03:35.102201+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:18.862009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094453099999999997,"stat_sum":{"num_bytes":180224,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":530,"num_write":2,"num_write_kb":176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9,8,4,5,6,10],"acting":[0,1,9,8,4,5,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"5","length":"2"},{"start":"8","length":"3"},{"start":"d","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":66,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-04-19T02:03:34.050625+0000","last_change":"2024-04-19T02:03:02.570839+0000","last_active":"2024-04-19T02:03:34.050625+0000","last_peered":"2024-04-19T02:03:34.050625+0000","last_clean":"2024-04-19T02:03:34.050625+0000","last_became_active":"2024-04-19T02:03:02.570116+0000","last_became_peered":"2024-04-19T02:03:02.570116+0000","last_unstale":"2024-04-19T02:03:34.050625+0000","last_undegraded":"2024-04-19T02:03:34.050625+0000","last_fullsized":"2024-04-19T02:03:34.050625+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:23.134425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":63848448,"num_objects":50,"num_object_clones":6,"num_object_copies":400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":2,"num_read":281,"num_read_kb":90583,"num_write":444,"num_write_kb":79968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":125747200,"data_stored":127287296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":539,"ondisk_log_size":539,"up":128,"acting":128,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":19,"seq":81604378635,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38908,"kb_used_data":7720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332932,"statfs":{"total":96636764160,"available":96596922368,"internally_reserved":0,"allocated":7905280,"data_stored":7665009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":19,"seq":81604378634,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40276,"kb_used_data":9088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331564,"statfs":{"total":96636764160,"available":96595521536,"internally_reserved":0,"allocated":9306112,"data_stored":9258353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411338,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45416,"kb_used_data":14236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326424,"statfs":{"total":96636764160,"available":96590258176,"internally_reserved":0,"allocated":14577664,"data_stored":14337393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411339,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44040,"kb_used_data":12860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327800,"statfs":{"total":96636764160,"available":96591667200,"internally_reserved":0,"allocated":13168640,"data_stored":13120881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411339,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38512,"kb_used_data":11484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94333328,"statfs":{"total":96636764160,"available":96597327872,"internally_reserved":0,"allocated":11759616,"data_stored":11708305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411339,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39744,"kb_used_data":12720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94332096,"statfs":{"total":96636764160,"available":96596066304,"internally_reserved":0,"allocated":13025280,"data_stored":12981400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":18,"seq":77309411337,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31484,"kb_used_data":4460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340356,"statfs":{"total":96636764160,"available":96604524544,"internally_reserved":0,"allocated":4567040,"data_stored":4523160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411338,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37644,"kb_used_data":10620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334196,"statfs":{"total":96636764160,"available":96598216704,"internally_reserved":0,"allocated":10874880,"data_stored":10831000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":18,"seq":77309411338,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38096,"kb_used_data":11072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94333744,"statfs":{"total":96636764160,"available":96597753856,"internally_reserved":0,"allocated":11337728,"data_stored":11293848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":17,"seq":73014444042,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35616,"kb_used_data":8596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336224,"statfs":{"total":96636764160,"available":96600293376,"internally_reserved":0,"allocated":8802304,"data_stored":8762303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":17,"seq":73014444042,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44504,"kb_used_data":13324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327336,"statfs":{"total":96636764160,"available":96591192064,"internally_reserved":0,"allocated":13643776,"data_stored":13403832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":17,"seq":73014444042,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42012,"kb_used_data":10824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329828,"statfs":{"total":96636764160,"available":96593743872,"internally_reserved":0,"allocated":11083776,"data_stored":10847384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8753152,"data_stored":8945664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10543104,"data_stored":10543104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7352320,"data_stored":7352320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11911168,"data_stored":12103680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11448320,"data_stored":11640832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12570624,"data_stored":12570624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4026368,"data_stored":4218880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":13180928,"data_stored":13373440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":10674176,"data_stored":10866688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":12615680,"data_stored":12808192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":8646656,"data_stored":8839168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":14024704,"data_stored":14024704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T02:03:39.057 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pgp_num 16 2024-04-19T02:03:39.081 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-04-19T02:03:39.081 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T02:03:39.086 INFO:tasks.rados.rados.0.smithi059.stdout:222: writing smithi05930959-41 from 3293184 to 3899392 tid 1 2024-04-19T02:03:39.089 INFO:tasks.rados.rados.0.smithi059.stdout:222: writing smithi05930959-41 from 3899392 to 4587520 tid 2 2024-04-19T02:03:39.094 INFO:tasks.rados.rados.0.smithi059.stdout:222: writing smithi05930959-41 from 4587520 to 5062656 tid 3 2024-04-19T02:03:39.094 INFO:tasks.rados.rados.0.smithi059.stdout:214: finishing copy_from racing read to smithi05930959-21 2024-04-19T02:03:39.094 INFO:tasks.rados.rados.0.smithi059.stdout:214: finishing copy_from to smithi05930959-21 2024-04-19T02:03:39.094 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 21 v 25 (ObjNum 72 snap 6 seq_num 72) dirty exists 2024-04-19T02:03:39.094 INFO:tasks.rados.rados.0.smithi059.stdout:215: finishing write tid 1 to smithi05930959-3 2024-04-19T02:03:39.094 INFO:tasks.rados.rados.0.smithi059.stdout:215: finishing write tid 2 to smithi05930959-3 2024-04-19T02:03:39.094 INFO:tasks.rados.rados.0.smithi059.stdout:215: finishing write tid 3 to smithi05930959-3 2024-04-19T02:03:39.094 INFO:tasks.rados.rados.0.smithi059.stdout:215: finishing write tid 4 to smithi05930959-3 2024-04-19T02:03:39.094 INFO:tasks.rados.rados.0.smithi059.stdout:215: finishing write tid 5 to smithi05930959-3 2024-04-19T02:03:39.095 INFO:tasks.rados.rados.0.smithi059.stdout:215: oid 3 updating version 0 to 51 2024-04-19T02:03:39.095 INFO:tasks.rados.rados.0.smithi059.stdout:215: oid 3 version 51 is already newer than 50 2024-04-19T02:03:39.095 INFO:tasks.rados.rados.0.smithi059.stdout:215: oid 3 updating version 51 to 52 2024-04-19T02:03:39.095 INFO:tasks.rados.rados.0.smithi059.stdout:215: oid 3 updating version 52 to 53 2024-04-19T02:03:39.095 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 3 v 53 (ObjNum 104 snap 14 seq_num 104) dirty exists 2024-04-19T02:03:39.095 INFO:tasks.rados.rados.0.smithi059.stdout:215: left oid 3 (ObjNum 104 snap 14 seq_num 104) 2024-04-19T02:03:39.095 INFO:tasks.rados.rados.0.smithi059.stdout:221: expect (ObjNum 31 snap 0 seq_num 31) 2024-04-19T02:03:39.102 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-04-19T02:03:39.102 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T02:03:39.110 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:39.110+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:39.111 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 44 v 54 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-19T02:03:39.111 INFO:tasks.rados.rados.0.smithi059.stdout:220: expect (ObjNum 56 snap 1 seq_num 56) 2024-04-19T02:03:39.125 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:39.126 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-04-19T02:03:39.126 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-04-19T02:03:39.126 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-04-19T02:03:39.126 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.126 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:237 3.2s0 3:407ecba7:::smithi05930959-22:2 [read 0~360448,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-19T02:03:39.126 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.646438+0000", 2024-04-19T02:03:39.126 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 22.480388405999999, 2024-04-19T02:03:39.126 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.024327603, 2024-04-19T02:03:39.126 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:39.126 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:39.126 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:39.126 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:39.126 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:39.127 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:39.127 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 237 2024-04-19T02:03:39.127 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.127 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:39.127 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.127 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:39.127 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646438+0000", 2024-04-19T02:03:39.127 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.127 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.127 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.127 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:39.127 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646438+0000", 2024-04-19T02:03:39.127 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.127 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.127 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.127 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:39.127 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646440+0000", 2024-04-19T02:03:39.128 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.254e-06 2024-04-19T02:03:39.128 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.128 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.128 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:39.128 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646447+0000", 2024-04-19T02:03:39.128 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.3289999999999996e-06 2024-04-19T02:03:39.128 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.128 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.128 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:39.128 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646463+0000", 2024-04-19T02:03:39.128 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5846e-05 2024-04-19T02:03:39.128 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.128 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.128 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.128 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646472+0000", 2024-04-19T02:03:39.128 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.2939999999999994e-06 2024-04-19T02:03:39.129 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.129 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.129 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:39.129 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646953+0000", 2024-04-19T02:03:39.129 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000480901 2024-04-19T02:03:39.129 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.129 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.129 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:03:39.129 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646971+0000", 2024-04-19T02:03:39.129 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8167000000000001e-05 2024-04-19T02:03:39.129 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.129 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.129 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:39.129 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.656667+0000", 2024-04-19T02:03:39.129 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0096952449999999999 2024-04-19T02:03:39.129 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.130 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.130 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:39.130 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.656833+0000", 2024-04-19T02:03:39.130 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00016697500000000001 2024-04-19T02:03:39.130 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.130 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.130 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:39.130 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.670766+0000", 2024-04-19T02:03:39.130 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.013932592000000001 2024-04-19T02:03:39.130 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.130 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:39.130 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.130 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.130 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.131 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:238 3.2s0 3:407ecba7:::smithi05930959-22:2 [read 0~360448,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-19T02:03:39.131 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.646510+0000", 2024-04-19T02:03:39.131 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 22.480317123999999, 2024-04-19T02:03:39.131 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.028744658999999999, 2024-04-19T02:03:39.131 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:39.131 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:39.131 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:39.131 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:39.131 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:39.131 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:39.131 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 238 2024-04-19T02:03:39.131 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.131 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:39.131 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.131 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:39.131 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646510+0000", 2024-04-19T02:03:39.131 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.132 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.132 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.132 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:39.132 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646510+0000", 2024-04-19T02:03:39.132 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.132 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.132 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.132 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:39.132 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646511+0000", 2024-04-19T02:03:39.132 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.8599999999999997e-07 2024-04-19T02:03:39.132 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.132 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.132 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:39.132 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646517+0000", 2024-04-19T02:03:39.132 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.1700000000000002e-06 2024-04-19T02:03:39.133 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.133 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.133 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:39.133 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646528+0000", 2024-04-19T02:03:39.133 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1202999999999999e-05 2024-04-19T02:03:39.133 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.133 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.133 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.133 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646537+0000", 2024-04-19T02:03:39.133 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7460000000000006e-06 2024-04-19T02:03:39.133 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.133 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.133 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:39.133 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.647105+0000", 2024-04-19T02:03:39.133 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00056801599999999996 2024-04-19T02:03:39.133 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.134 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.134 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:03:39.134 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.647127+0000", 2024-04-19T02:03:39.134 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.175e-05 2024-04-19T02:03:39.134 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.134 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.134 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:39.134 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.657504+0000", 2024-04-19T02:03:39.134 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010377961 2024-04-19T02:03:39.134 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.134 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.134 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:39.134 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.657826+0000", 2024-04-19T02:03:39.134 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00032128300000000001 2024-04-19T02:03:39.134 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.135 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.135 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:39.135 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.675254+0000", 2024-04-19T02:03:39.135 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.017428644 2024-04-19T02:03:39.135 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.135 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:39.135 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.135 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.135 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.135 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:239 3.2s0 3:407ecba7:::smithi05930959-22:2 [read 0~360448,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-19T02:03:39.135 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.646549+0000", 2024-04-19T02:03:39.135 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 22.480277653000002, 2024-04-19T02:03:39.135 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.029381570999999999, 2024-04-19T02:03:39.135 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:39.135 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:39.135 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:39.136 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:39.136 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:39.136 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:39.136 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 239 2024-04-19T02:03:39.136 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.136 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:39.136 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.136 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:39.136 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646549+0000", 2024-04-19T02:03:39.136 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.136 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.136 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.136 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:39.136 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646549+0000", 2024-04-19T02:03:39.136 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.136 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.136 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.137 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:39.137 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646550+0000", 2024-04-19T02:03:39.137 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8400000000000004e-07 2024-04-19T02:03:39.137 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.137 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.137 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:39.137 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646555+0000", 2024-04-19T02:03:39.137 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9989999999999999e-06 2024-04-19T02:03:39.137 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.137 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.137 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:39.137 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646567+0000", 2024-04-19T02:03:39.137 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.202e-05 2024-04-19T02:03:39.137 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.138 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.138 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.138 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646575+0000", 2024-04-19T02:03:39.138 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.7640000000000005e-06 2024-04-19T02:03:39.138 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.138 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.138 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:39.138 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.647229+0000", 2024-04-19T02:03:39.138 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00065415299999999998 2024-04-19T02:03:39.138 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.138 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.138 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:03:39.138 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.647255+0000", 2024-04-19T02:03:39.138 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5795999999999998e-05 2024-04-19T02:03:39.139 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.139 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.139 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:39.139 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.659835+0000", 2024-04-19T02:03:39.139 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012580713 2024-04-19T02:03:39.139 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.139 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.139 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:39.139 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.660090+0000", 2024-04-19T02:03:39.139 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00025430899999999998 2024-04-19T02:03:39.139 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.139 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.139 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:39.139 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.675931+0000", 2024-04-19T02:03:39.139 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.015841133 2024-04-19T02:03:39.139 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.140 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:39.140 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.140 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.140 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.140 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:349 3.2s0 3:43b7722e:::smithi05930959-10:head [copy-from ver 11] snapc 8=[8,6,5] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:39.140 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.746333+0000", 2024-04-19T02:03:39.140 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 14.380493496, 2024-04-19T02:03:39.140 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.034219922999999999, 2024-04-19T02:03:39.140 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:39.140 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:39.140 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:39.140 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:39.141 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:39.141 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:39.141 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 349 2024-04-19T02:03:39.141 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.141 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:39.141 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.141 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:39.141 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746333+0000", 2024-04-19T02:03:39.141 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.141 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.141 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.141 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:39.141 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746333+0000", 2024-04-19T02:03:39.141 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.142 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.142 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.142 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:39.142 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746337+0000", 2024-04-19T02:03:39.142 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4769999999999999e-06 2024-04-19T02:03:39.142 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.142 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.142 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:39.142 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746352+0000", 2024-04-19T02:03:39.142 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5277e-05 2024-04-19T02:03:39.142 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.142 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.142 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:39.142 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746387+0000", 2024-04-19T02:03:39.143 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4436e-05 2024-04-19T02:03:39.143 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.143 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.143 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.143 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746403+0000", 2024-04-19T02:03:39.143 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6475999999999999e-05 2024-04-19T02:03:39.143 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.143 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.143 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:39.143 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746490+0000", 2024-04-19T02:03:39.143 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7213000000000007e-05 2024-04-19T02:03:39.143 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.144 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.144 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "op must wait for map", 2024-04-19T02:03:39.144 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746497+0000", 2024-04-19T02:03:39.144 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.0520000000000004e-06 2024-04-19T02:03:39.144 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.144 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.144 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:39.144 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749865+0000", 2024-04-19T02:03:39.144 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0033676069999999999 2024-04-19T02:03:39.144 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.144 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.144 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:39.144 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749938+0000", 2024-04-19T02:03:39.144 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.3530999999999998e-05 2024-04-19T02:03:39.144 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.145 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.145 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:39.145 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.768855+0000", 2024-04-19T02:03:39.145 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.018916493 2024-04-19T02:03:39.145 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.145 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.145 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:39.145 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.771377+0000", 2024-04-19T02:03:39.145 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0025221520000000002 2024-04-19T02:03:39.145 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.145 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.145 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:39.145 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.780460+0000", 2024-04-19T02:03:39.145 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0090830389999999993 2024-04-19T02:03:39.145 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.145 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.146 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:39.146 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.780553+0000", 2024-04-19T02:03:39.146 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3170000000000004e-05 2024-04-19T02:03:39.146 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.146 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:39.146 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.146 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.146 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.146 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:350 3.2s0 3:43b7722e:::smithi05930959-10:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:39.146 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.746450+0000", 2024-04-19T02:03:39.146 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 14.380376837, 2024-04-19T02:03:39.146 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.024575331999999998, 2024-04-19T02:03:39.147 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:39.147 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:39.147 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:39.147 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:39.147 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:39.147 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:39.147 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 350 2024-04-19T02:03:39.147 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.147 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:39.147 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.147 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:39.148 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746450+0000", 2024-04-19T02:03:39.148 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.148 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.148 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.148 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:39.148 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746450+0000", 2024-04-19T02:03:39.148 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.148 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.148 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.148 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:39.148 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746451+0000", 2024-04-19T02:03:39.148 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0400000000000005e-07 2024-04-19T02:03:39.148 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.149 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.149 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:39.149 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746456+0000", 2024-04-19T02:03:39.149 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5199999999999997e-06 2024-04-19T02:03:39.149 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.149 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.149 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:39.149 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746470+0000", 2024-04-19T02:03:39.149 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3935e-05 2024-04-19T02:03:39.149 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.149 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.149 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.149 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746481+0000", 2024-04-19T02:03:39.150 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0919e-05 2024-04-19T02:03:39.150 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.150 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.150 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:39.150 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746552+0000", 2024-04-19T02:03:39.150 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.0235999999999996e-05 2024-04-19T02:03:39.150 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.150 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.150 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:03:39.150 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746557+0000", 2024-04-19T02:03:39.150 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.6570000000000002e-06 2024-04-19T02:03:39.150 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.151 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.151 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:39.151 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750143+0000", 2024-04-19T02:03:39.151 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0035860549999999999 2024-04-19T02:03:39.151 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.151 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.151 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for blocked object", 2024-04-19T02:03:39.151 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750177+0000", 2024-04-19T02:03:39.151 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3796e-05 2024-04-19T02:03:39.151 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.151 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.151 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:39.151 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.770842+0000", 2024-04-19T02:03:39.151 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.020664657999999999 2024-04-19T02:03:39.152 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.152 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.152 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:39.152 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.770923+0000", 2024-04-19T02:03:39.152 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0869000000000001e-05 2024-04-19T02:03:39.152 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.152 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.152 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:39.152 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.771025+0000", 2024-04-19T02:03:39.152 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010278299999999999 2024-04-19T02:03:39.152 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.152 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:39.152 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.152 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.152 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.153 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:422 3.2s0 3:43b7722e:::smithi05930959-10:head [copy-from ver 11] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:39.153 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.193952+0000", 2024-04-19T02:03:39.153 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 8.9328744980000003, 2024-04-19T02:03:39.153 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.044114190999999997, 2024-04-19T02:03:39.153 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:39.153 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:39.153 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:39.153 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:39.153 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:39.153 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:39.153 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 422 2024-04-19T02:03:39.153 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.153 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:39.154 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.154 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:39.154 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.193952+0000", 2024-04-19T02:03:39.154 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.154 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.154 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.154 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:39.154 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.193952+0000", 2024-04-19T02:03:39.154 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.154 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.154 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.154 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:39.154 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.193955+0000", 2024-04-19T02:03:39.155 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3159999999999999e-06 2024-04-19T02:03:39.155 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.155 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.155 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:39.155 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.193968+0000", 2024-04-19T02:03:39.155 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3726999999999999e-05 2024-04-19T02:03:39.155 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.155 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.155 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:39.155 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.193993+0000", 2024-04-19T02:03:39.155 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4499e-05 2024-04-19T02:03:39.155 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.155 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.156 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.156 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194005+0000", 2024-04-19T02:03:39.156 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1770000000000001e-05 2024-04-19T02:03:39.156 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.156 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.156 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:39.156 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194109+0000", 2024-04-19T02:03:39.156 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000103904 2024-04-19T02:03:39.156 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.156 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.156 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:39.156 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194249+0000", 2024-04-19T02:03:39.156 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014036699999999999 2024-04-19T02:03:39.157 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.157 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.157 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:39.157 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.224308+0000", 2024-04-19T02:03:39.157 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.030059210999999999 2024-04-19T02:03:39.157 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.157 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.157 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:39.157 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.237172+0000", 2024-04-19T02:03:39.157 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012863734999999999 2024-04-19T02:03:39.157 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.157 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.158 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:39.158 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.237230+0000", 2024-04-19T02:03:39.158 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.8238e-05 2024-04-19T02:03:39.158 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.158 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.158 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:39.158 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.238067+0000", 2024-04-19T02:03:39.158 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000836424 2024-04-19T02:03:39.158 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.158 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:39.158 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.158 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.158 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.158 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:423 3.2s0 3:43b7722e:::smithi05930959-10:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:39.159 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194040+0000", 2024-04-19T02:03:39.159 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 8.9327871680000008, 2024-04-19T02:03:39.159 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.040466800999999997, 2024-04-19T02:03:39.159 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:39.159 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:39.159 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:39.159 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:39.159 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:39.159 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:39.159 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 423 2024-04-19T02:03:39.159 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.159 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:39.159 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.159 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:39.159 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194040+0000", 2024-04-19T02:03:39.160 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.160 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.160 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.160 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:39.160 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194040+0000", 2024-04-19T02:03:39.160 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.160 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.160 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.160 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:39.160 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194040+0000", 2024-04-19T02:03:39.160 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.4700000000000001e-07 2024-04-19T02:03:39.160 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.160 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.160 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:39.161 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194044+0000", 2024-04-19T02:03:39.161 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9049999999999999e-06 2024-04-19T02:03:39.161 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.161 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.161 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:39.161 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194056+0000", 2024-04-19T02:03:39.161 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1567e-05 2024-04-19T02:03:39.161 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.161 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.161 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.161 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194063+0000", 2024-04-19T02:03:39.161 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.1539999999999996e-06 2024-04-19T02:03:39.161 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.161 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.162 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:39.162 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194511+0000", 2024-04-19T02:03:39.162 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00044832999999999999 2024-04-19T02:03:39.162 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.162 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.162 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for blocked object", 2024-04-19T02:03:39.162 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194561+0000", 2024-04-19T02:03:39.162 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9336999999999999e-05 2024-04-19T02:03:39.162 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.162 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.162 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:39.162 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.234276+0000", 2024-04-19T02:03:39.163 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.039715156000000001 2024-04-19T02:03:39.163 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.163 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.163 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:39.163 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.234405+0000", 2024-04-19T02:03:39.163 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00012947100000000001 2024-04-19T02:03:39.163 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.163 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.163 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:39.163 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.234507+0000", 2024-04-19T02:03:39.163 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000101234 2024-04-19T02:03:39.163 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.163 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:39.163 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.164 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.164 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.164 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:424 3.2s0 3:407ecba7:::smithi05930959-22:head [append 0~0] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:39.164 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194090+0000", 2024-04-19T02:03:39.164 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 8.9327364229999997, 2024-04-19T02:03:39.164 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.04103888, 2024-04-19T02:03:39.164 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:39.164 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:39.164 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:39.164 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:39.164 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:39.164 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:39.164 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 424 2024-04-19T02:03:39.164 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.164 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:39.165 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.165 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:39.165 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194090+0000", 2024-04-19T02:03:39.165 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.165 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.165 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.165 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:39.165 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194090+0000", 2024-04-19T02:03:39.165 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.165 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.165 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.165 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:39.165 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194091+0000", 2024-04-19T02:03:39.165 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.9599999999999999e-07 2024-04-19T02:03:39.165 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.165 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.165 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:39.166 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194095+0000", 2024-04-19T02:03:39.166 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.019e-06 2024-04-19T02:03:39.166 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.166 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.166 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:39.166 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194104+0000", 2024-04-19T02:03:39.166 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.6030000000000005e-06 2024-04-19T02:03:39.166 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.166 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.166 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.166 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194110+0000", 2024-04-19T02:03:39.166 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.6880000000000002e-06 2024-04-19T02:03:39.166 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.166 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.167 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:39.167 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194597+0000", 2024-04-19T02:03:39.167 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00048649499999999998 2024-04-19T02:03:39.167 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.167 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.167 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:39.167 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194669+0000", 2024-04-19T02:03:39.167 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.2081000000000003e-05 2024-04-19T02:03:39.167 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.167 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.167 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:39.167 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195298+0000", 2024-04-19T02:03:39.167 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000628635 2024-04-19T02:03:39.167 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.168 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.168 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:39.168 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.201226+0000", 2024-04-19T02:03:39.168 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0059289199999999998 2024-04-19T02:03:39.168 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.168 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.168 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:39.168 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235058+0000", 2024-04-19T02:03:39.168 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.033831539000000001 2024-04-19T02:03:39.168 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.168 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.168 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:39.168 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235129+0000", 2024-04-19T02:03:39.168 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.1303999999999994e-05 2024-04-19T02:03:39.169 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.169 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:39.169 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.169 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.169 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.169 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:425 3.2s0 3:407ecba7:::smithi05930959-22:head [rollback a] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:39.169 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194118+0000", 2024-04-19T02:03:39.169 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 8.9327089599999994, 2024-04-19T02:03:39.169 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.041424615999999997, 2024-04-19T02:03:39.169 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:39.169 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:39.169 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:39.169 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:39.169 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:39.169 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:39.170 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 425 2024-04-19T02:03:39.170 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.170 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:39.170 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.170 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:39.170 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194118+0000", 2024-04-19T02:03:39.170 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.170 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.170 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.170 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:39.170 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194118+0000", 2024-04-19T02:03:39.170 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.170 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.170 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.171 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:39.171 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194118+0000", 2024-04-19T02:03:39.171 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7799999999999997e-07 2024-04-19T02:03:39.171 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.171 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.171 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:39.171 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194120+0000", 2024-04-19T02:03:39.171 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9740000000000001e-06 2024-04-19T02:03:39.171 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.171 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.171 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:39.171 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194124+0000", 2024-04-19T02:03:39.171 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1540000000000004e-06 2024-04-19T02:03:39.171 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.172 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.172 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.172 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194127+0000", 2024-04-19T02:03:39.172 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1379999999999999e-06 2024-04-19T02:03:39.172 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.172 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.172 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:39.172 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195791+0000", 2024-04-19T02:03:39.172 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0016630779999999999 2024-04-19T02:03:39.172 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.172 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.172 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:39.172 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195884+0000", 2024-04-19T02:03:39.172 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3885e-05 2024-04-19T02:03:39.173 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.173 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.173 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:39.173 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.196827+0000", 2024-04-19T02:03:39.173 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00094285400000000002 2024-04-19T02:03:39.173 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.173 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.173 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:39.173 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.201229+0000", 2024-04-19T02:03:39.173 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0044017359999999998 2024-04-19T02:03:39.173 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.173 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.174 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:39.174 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235479+0000", 2024-04-19T02:03:39.174 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.034249891999999997 2024-04-19T02:03:39.174 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.174 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.174 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:39.174 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235543+0000", 2024-04-19T02:03:39.174 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.3627000000000006e-05 2024-04-19T02:03:39.174 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.174 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:39.174 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.174 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.174 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.174 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:426 3.2s0 3:407ecba7:::smithi05930959-22:head [append 0~0] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:39.175 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194132+0000", 2024-04-19T02:03:39.175 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 8.9326953489999994, 2024-04-19T02:03:39.175 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.041767809000000003, 2024-04-19T02:03:39.175 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:39.175 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:39.175 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:39.175 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:39.175 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:39.175 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:39.175 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 426 2024-04-19T02:03:39.175 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.175 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:39.175 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.175 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:39.175 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194132+0000", 2024-04-19T02:03:39.175 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.176 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.176 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.176 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:39.176 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194132+0000", 2024-04-19T02:03:39.176 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.176 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.176 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.176 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:39.176 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194132+0000", 2024-04-19T02:03:39.176 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5199999999999998e-07 2024-04-19T02:03:39.176 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.176 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.176 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:39.176 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194133+0000", 2024-04-19T02:03:39.176 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.624e-06 2024-04-19T02:03:39.177 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.177 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.177 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:39.177 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194137+0000", 2024-04-19T02:03:39.177 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3550000000000001e-06 2024-04-19T02:03:39.177 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.177 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.177 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.177 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194140+0000", 2024-04-19T02:03:39.177 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2229999999999999e-06 2024-04-19T02:03:39.177 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.177 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.177 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:39.177 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.198324+0000", 2024-04-19T02:03:39.177 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0041834799999999998 2024-04-19T02:03:39.178 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.178 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.178 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:39.178 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.198430+0000", 2024-04-19T02:03:39.178 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010597100000000001 2024-04-19T02:03:39.178 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.178 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.178 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:39.178 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.199339+0000", 2024-04-19T02:03:39.178 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00090987100000000003 2024-04-19T02:03:39.178 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.178 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.178 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:39.179 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.201230+0000", 2024-04-19T02:03:39.179 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0018903170000000001 2024-04-19T02:03:39.179 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.179 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.179 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:39.179 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235836+0000", 2024-04-19T02:03:39.179 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.034606382999999998 2024-04-19T02:03:39.179 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.179 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.179 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:39.179 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235899+0000", 2024-04-19T02:03:39.179 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.3232999999999996e-05 2024-04-19T02:03:39.179 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.179 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:39.180 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.180 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.180 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.180 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:437 3.bs0 3:d6e3a590:::smithi05930959-43:head [copy-from ver 51] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:39.180 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194173+0000", 2024-04-19T02:03:39.180 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 8.9326541030000008, 2024-04-19T02:03:39.180 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.054675901999999998, 2024-04-19T02:03:39.180 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:39.180 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:39.180 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:39.180 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:39.180 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:39.180 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:39.180 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 437 2024-04-19T02:03:39.181 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.181 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:39.181 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.181 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:39.181 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194173+0000", 2024-04-19T02:03:39.181 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.181 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.181 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.181 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:39.181 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194173+0000", 2024-04-19T02:03:39.181 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.181 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.181 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.181 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:39.182 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194173+0000", 2024-04-19T02:03:39.182 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1100000000000002e-07 2024-04-19T02:03:39.182 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.182 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.182 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:39.182 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194179+0000", 2024-04-19T02:03:39.182 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.122e-06 2024-04-19T02:03:39.182 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.182 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.182 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:39.182 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194185+0000", 2024-04-19T02:03:39.182 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.3109999999999996e-06 2024-04-19T02:03:39.182 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.182 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.183 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.183 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194188+0000", 2024-04-19T02:03:39.183 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0230000000000001e-06 2024-04-19T02:03:39.183 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.183 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.183 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:39.183 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194246+0000", 2024-04-19T02:03:39.183 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.8865e-05 2024-04-19T02:03:39.183 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.183 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.183 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:39.183 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194356+0000", 2024-04-19T02:03:39.183 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000109558 2024-04-19T02:03:39.183 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.184 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.184 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:39.184 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.240077+0000", 2024-04-19T02:03:39.184 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.045721108000000003 2024-04-19T02:03:39.184 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.184 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.184 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:39.184 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.244667+0000", 2024-04-19T02:03:39.184 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0045899000000000001 2024-04-19T02:03:39.184 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.184 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.185 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:39.185 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.248766+0000", 2024-04-19T02:03:39.185 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0040986579999999998 2024-04-19T02:03:39.185 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.185 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.185 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:39.185 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.248849+0000", 2024-04-19T02:03:39.185 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3046000000000004e-05 2024-04-19T02:03:39.185 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.185 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:39.185 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.185 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.185 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.186 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:438 3.bs0 3:d6e3a590:::smithi05930959-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:39.186 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194196+0000", 2024-04-19T02:03:39.186 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 8.9326312899999998, 2024-04-19T02:03:39.186 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.047992689999999998, 2024-04-19T02:03:39.186 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:39.186 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:39.186 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:39.186 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:39.186 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:39.186 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:39.186 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 438 2024-04-19T02:03:39.186 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.186 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:39.187 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.187 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:39.187 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194196+0000", 2024-04-19T02:03:39.187 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.187 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.187 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.187 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:39.187 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194196+0000", 2024-04-19T02:03:39.187 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.187 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.187 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.187 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:39.187 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194196+0000", 2024-04-19T02:03:39.187 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4700000000000002e-07 2024-04-19T02:03:39.187 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.187 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.188 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:39.188 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194198+0000", 2024-04-19T02:03:39.188 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4320000000000002e-06 2024-04-19T02:03:39.188 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.188 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.188 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:39.188 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194204+0000", 2024-04-19T02:03:39.188 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0850000000000002e-06 2024-04-19T02:03:39.188 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.188 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.188 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.188 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194209+0000", 2024-04-19T02:03:39.188 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5090000000000004e-06 2024-04-19T02:03:39.188 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.188 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.188 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:39.189 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194533+0000", 2024-04-19T02:03:39.189 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00032412499999999998 2024-04-19T02:03:39.189 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.189 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.189 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for blocked object", 2024-04-19T02:03:39.189 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194593+0000", 2024-04-19T02:03:39.189 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.9854e-05 2024-04-19T02:03:39.189 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.189 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.189 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:39.189 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.242048+0000", 2024-04-19T02:03:39.189 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.047455152 2024-04-19T02:03:39.189 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.189 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.189 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:39.189 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.242113+0000", 2024-04-19T02:03:39.190 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.4573000000000007e-05 2024-04-19T02:03:39.190 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.190 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.190 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:39.190 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.242188+0000", 2024-04-19T02:03:39.190 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.5612999999999996e-05 2024-04-19T02:03:39.190 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.190 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:39.190 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.190 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.190 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.190 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:439 3.2s0 3:43ce42f3:::smithi05930959-27:head [append 0~0] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:39.190 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194231+0000", 2024-04-19T02:03:39.190 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 8.9325958570000008, 2024-04-19T02:03:39.191 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.041997487, 2024-04-19T02:03:39.191 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:39.191 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:39.191 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:39.191 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:39.191 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:39.191 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:39.191 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 439 2024-04-19T02:03:39.191 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.191 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:39.191 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.191 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:39.191 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194231+0000", 2024-04-19T02:03:39.192 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.192 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.192 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.192 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:39.192 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194231+0000", 2024-04-19T02:03:39.192 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.192 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.192 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.192 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:39.192 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194232+0000", 2024-04-19T02:03:39.192 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7999999999999995e-07 2024-04-19T02:03:39.192 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.192 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.192 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:39.193 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194236+0000", 2024-04-19T02:03:39.193 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2139999999999998e-06 2024-04-19T02:03:39.193 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.193 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.193 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:39.193 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194243+0000", 2024-04-19T02:03:39.193 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.5979999999999999e-06 2024-04-19T02:03:39.193 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.193 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.193 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.193 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194248+0000", 2024-04-19T02:03:39.193 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.3370000000000001e-06 2024-04-19T02:03:39.193 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.194 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.194 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:39.194 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.204081+0000", 2024-04-19T02:03:39.194 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0098332009999999997 2024-04-19T02:03:39.194 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.194 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.194 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:39.194 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.204143+0000", 2024-04-19T02:03:39.194 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.156e-05 2024-04-19T02:03:39.194 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.194 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.194 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:39.194 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.204875+0000", 2024-04-19T02:03:39.194 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00073241099999999998 2024-04-19T02:03:39.195 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.195 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.195 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:39.195 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.208933+0000", 2024-04-19T02:03:39.195 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0040575990000000003 2024-04-19T02:03:39.198 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.198 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.198 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:39.198 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.236159+0000", 2024-04-19T02:03:39.198 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.027226274000000002 2024-04-19T02:03:39.198 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.198 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.198 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:39.198 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.236229+0000", 2024-04-19T02:03:39.198 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9713000000000002e-05 2024-04-19T02:03:39.198 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.199 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:39.199 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.199 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.199 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.199 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:440 3.2s0 3:43ce42f3:::smithi05930959-27:head [rollback a] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:39.199 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194253+0000", 2024-04-19T02:03:39.199 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 8.9325736169999992, 2024-04-19T02:03:39.199 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.042233729999999997, 2024-04-19T02:03:39.199 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:39.199 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:39.199 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:39.199 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:39.199 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:39.199 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:39.200 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 440 2024-04-19T02:03:39.200 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.200 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:39.200 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.200 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:39.200 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194253+0000", 2024-04-19T02:03:39.200 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.200 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.200 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.200 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:39.200 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194253+0000", 2024-04-19T02:03:39.200 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.200 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.200 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.201 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:39.201 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194254+0000", 2024-04-19T02:03:39.201 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8299999999999998e-07 2024-04-19T02:03:39.201 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.201 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.201 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:39.201 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194255+0000", 2024-04-19T02:03:39.201 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7120000000000001e-06 2024-04-19T02:03:39.201 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.201 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.201 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:39.201 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194259+0000", 2024-04-19T02:03:39.201 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.517e-06 2024-04-19T02:03:39.201 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.202 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.202 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.202 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194261+0000", 2024-04-19T02:03:39.202 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.689e-06 2024-04-19T02:03:39.202 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.202 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.202 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:39.202 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.205980+0000", 2024-04-19T02:03:39.202 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.011718561000000001 2024-04-19T02:03:39.202 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.202 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.202 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:39.202 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.206074+0000", 2024-04-19T02:03:39.202 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3556999999999999e-05 2024-04-19T02:03:39.203 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.203 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.203 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:39.203 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.206673+0000", 2024-04-19T02:03:39.203 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00059902100000000004 2024-04-19T02:03:39.203 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.203 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.203 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:39.203 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.208936+0000", 2024-04-19T02:03:39.203 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002262975 2024-04-19T02:03:39.203 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.203 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.203 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:39.203 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.236430+0000", 2024-04-19T02:03:39.204 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.027494338 2024-04-19T02:03:39.204 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.204 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.204 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:39.204 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.236487+0000", 2024-04-19T02:03:39.204 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7077000000000003e-05 2024-04-19T02:03:39.204 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.204 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:39.204 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.204 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.204 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.204 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:441 3.2s0 3:43ce42f3:::smithi05930959-27:head [append 0~0] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:39.204 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194270+0000", 2024-04-19T02:03:39.204 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 8.9325571349999997, 2024-04-19T02:03:39.205 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.042541760999999997, 2024-04-19T02:03:39.205 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:39.205 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:39.205 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:39.205 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:39.205 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:39.205 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:39.205 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 441 2024-04-19T02:03:39.205 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.205 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:39.205 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.205 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:39.205 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194270+0000", 2024-04-19T02:03:39.205 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.206 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.206 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.206 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:39.206 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194270+0000", 2024-04-19T02:03:39.206 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.206 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.206 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.206 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:39.206 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194270+0000", 2024-04-19T02:03:39.206 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2500000000000001e-07 2024-04-19T02:03:39.206 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.206 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.206 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:39.206 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194271+0000", 2024-04-19T02:03:39.207 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3939999999999999e-06 2024-04-19T02:03:39.207 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.207 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.207 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:39.207 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194275+0000", 2024-04-19T02:03:39.207 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.827e-06 2024-04-19T02:03:39.207 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.207 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.207 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.207 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194278+0000", 2024-04-19T02:03:39.207 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7889999999999999e-06 2024-04-19T02:03:39.207 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.207 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.207 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:39.208 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.207160+0000", 2024-04-19T02:03:39.208 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012881867999999999 2024-04-19T02:03:39.208 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.208 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.208 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:39.208 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.207262+0000", 2024-04-19T02:03:39.208 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000102142 2024-04-19T02:03:39.208 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.208 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.208 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:39.208 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.207787+0000", 2024-04-19T02:03:39.208 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00052537099999999995 2024-04-19T02:03:39.208 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.208 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.208 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:39.209 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.208937+0000", 2024-04-19T02:03:39.209 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00114908 2024-04-19T02:03:39.209 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.209 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.209 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:39.209 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.236712+0000", 2024-04-19T02:03:39.209 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.027775428000000001 2024-04-19T02:03:39.209 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.209 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.209 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:39.209 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.236811+0000", 2024-04-19T02:03:39.209 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9537000000000006e-05 2024-04-19T02:03:39.209 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.209 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:39.210 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.210 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.210 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.210 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(osd.7.18:2 3.2s0 3:465821b6:::smithi05930959-32:head [assert-version v14,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:39.210 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.199189+0000", 2024-04-19T02:03:39.210 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 8.9276379000000006, 2024-04-19T02:03:39.210 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.035632812, 2024-04-19T02:03:39.210 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:39.210 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:39.210 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:39.210 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:39.210 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.210 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:39.210 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.199189+0000", 2024-04-19T02:03:39.210 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.211 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.211 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.211 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:39.211 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.199189+0000", 2024-04-19T02:03:39.211 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.211 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.211 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.211 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:39.211 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.199190+0000", 2024-04-19T02:03:39.211 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5069999999999999e-06 2024-04-19T02:03:39.211 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.211 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.211 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:39.211 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.199198+0000", 2024-04-19T02:03:39.211 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.199e-06 2024-04-19T02:03:39.211 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.212 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.212 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:39.212 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.199211+0000", 2024-04-19T02:03:39.212 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3567000000000001e-05 2024-04-19T02:03:39.212 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.212 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.212 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.212 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.199220+0000", 2024-04-19T02:03:39.212 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2719999999999997e-06 2024-04-19T02:03:39.212 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.212 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.212 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:39.212 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.200485+0000", 2024-04-19T02:03:39.212 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001265195 2024-04-19T02:03:39.212 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.212 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.213 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:39.213 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.200596+0000", 2024-04-19T02:03:39.213 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00011098 2024-04-19T02:03:39.213 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.213 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.213 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:39.213 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.234822+0000", 2024-04-19T02:03:39.213 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.034226092 2024-04-19T02:03:39.213 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.213 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:39.213 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.213 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.213 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.213 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.2s0 46/26 ECSubReadReply(tid=166, attrs_read=0))", 2024-04-19T02:03:39.213 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.209587+0000", 2024-04-19T02:03:39.214 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 8.9172397150000009, 2024-04-19T02:03:39.214 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.025249772, 2024-04-19T02:03:39.214 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:39.214 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:39.214 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:39.214 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:39.214 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.214 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:39.214 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.209587+0000", 2024-04-19T02:03:39.214 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.214 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.214 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.215 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:39.215 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.209587+0000", 2024-04-19T02:03:39.215 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.215 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.215 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.215 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:39.215 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.209588+0000", 2024-04-19T02:03:39.215 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.4099999999999999e-07 2024-04-19T02:03:39.215 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.215 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.215 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:39.215 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.209834+0000", 2024-04-19T02:03:39.215 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024650200000000001 2024-04-19T02:03:39.215 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.216 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.216 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:39.216 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.209854+0000", 2024-04-19T02:03:39.216 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9758999999999998e-05 2024-04-19T02:03:39.216 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.216 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.216 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.216 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.209867+0000", 2024-04-19T02:03:39.216 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2917e-05 2024-04-19T02:03:39.216 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.216 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.216 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:39.216 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.234535+0000", 2024-04-19T02:03:39.217 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.024668606999999999 2024-04-19T02:03:39.217 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.217 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.217 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:39.217 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.234837+0000", 2024-04-19T02:03:39.217 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00030144599999999997 2024-04-19T02:03:39.217 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.217 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:39.217 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.217 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.217 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.217 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 46/26 ECSubWriteReply(tid=163, last_complete=46'78, committed=1, applied=1))", 2024-04-19T02:03:39.217 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.210911+0000", 2024-04-19T02:03:39.218 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 8.9159157229999995, 2024-04-19T02:03:39.218 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.024349902, 2024-04-19T02:03:39.218 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:39.218 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:39.218 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:39.218 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:39.218 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.218 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:39.218 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210911+0000", 2024-04-19T02:03:39.218 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.218 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.218 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.218 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:39.219 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210911+0000", 2024-04-19T02:03:39.219 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.219 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.219 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.219 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:39.219 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210912+0000", 2024-04-19T02:03:39.219 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8700000000000005e-07 2024-04-19T02:03:39.219 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.219 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.219 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:39.219 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210916+0000", 2024-04-19T02:03:39.219 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4340000000000002e-06 2024-04-19T02:03:39.219 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.220 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.220 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:39.220 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210933+0000", 2024-04-19T02:03:39.220 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7125999999999998e-05 2024-04-19T02:03:39.220 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.220 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.220 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.220 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210952+0000", 2024-04-19T02:03:39.220 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8320000000000001e-05 2024-04-19T02:03:39.220 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.221 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.221 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:39.221 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235246+0000", 2024-04-19T02:03:39.221 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.024294759999999999 2024-04-19T02:03:39.221 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.221 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.221 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:39.221 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235261+0000", 2024-04-19T02:03:39.221 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4575e-05 2024-04-19T02:03:39.221 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.221 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:39.221 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.222 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.222 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.222 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 46/26 ECSubWriteReply(tid=164, last_complete=46'79, committed=1, applied=1))", 2024-04-19T02:03:39.222 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.210983+0000", 2024-04-19T02:03:39.222 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 8.9158439079999994, 2024-04-19T02:03:39.222 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.024646257000000001, 2024-04-19T02:03:39.222 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:39.222 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:39.222 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:39.222 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:39.222 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.222 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:39.222 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210983+0000", 2024-04-19T02:03:39.223 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.223 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.223 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.223 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:39.223 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210983+0000", 2024-04-19T02:03:39.223 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.223 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.223 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.223 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:39.223 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210983+0000", 2024-04-19T02:03:39.223 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2100000000000002e-07 2024-04-19T02:03:39.223 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.223 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.223 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:39.223 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210987+0000", 2024-04-19T02:03:39.224 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2789999999999999e-06 2024-04-19T02:03:39.224 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.224 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.224 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:39.224 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210995+0000", 2024-04-19T02:03:39.224 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3790000000000006e-06 2024-04-19T02:03:39.224 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.224 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.224 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.224 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.211007+0000", 2024-04-19T02:03:39.224 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2373e-05 2024-04-19T02:03:39.224 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.224 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.224 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:39.224 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235615+0000", 2024-04-19T02:03:39.224 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.024607688999999999 2024-04-19T02:03:39.225 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.225 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.225 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:39.225 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235629+0000", 2024-04-19T02:03:39.225 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4116e-05 2024-04-19T02:03:39.225 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.225 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:39.225 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.225 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.225 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.225 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 46/26 ECSubWriteReply(tid=167, last_complete=46'81, committed=1, applied=1))", 2024-04-19T02:03:39.225 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.211022+0000", 2024-04-19T02:03:39.225 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 8.9158050230000008, 2024-04-19T02:03:39.225 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.024969574000000001, 2024-04-19T02:03:39.225 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:39.225 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:39.225 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:39.226 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:39.226 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.226 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:39.226 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.211022+0000", 2024-04-19T02:03:39.226 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.226 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.226 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.226 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:39.226 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.211022+0000", 2024-04-19T02:03:39.226 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:39.226 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.226 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.226 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:39.226 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.211022+0000", 2024-04-19T02:03:39.227 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2000000000000001e-07 2024-04-19T02:03:39.227 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.227 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.227 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:39.227 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.211025+0000", 2024-04-19T02:03:39.227 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.717e-06 2024-04-19T02:03:39.227 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.227 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.227 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:39.227 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.218661+0000", 2024-04-19T02:03:39.227 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0076366139999999999 2024-04-19T02:03:39.227 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.227 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.227 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.227 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.218669+0000", 2024-04-19T02:03:39.228 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.678e-06 2024-04-19T02:03:39.228 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.228 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.228 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:39.228 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235978+0000", 2024-04-19T02:03:39.228 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.017308430999999999 2024-04-19T02:03:39.228 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:39.228 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:39.228 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:39.228 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235991+0000", 2024-04-19T02:03:39.228 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3814e-05 2024-04-19T02:03:39.228 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.228 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:39.228 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.228 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:39.228 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:39.228 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:39.229 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-04-19T02:03:39.229 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T02:03:39.230 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-19T02:03:39.231 INFO:tasks.rados.rados.0.smithi059.stdout:222: finishing write tid 1 to smithi05930959-41 2024-04-19T02:03:39.231 INFO:tasks.rados.rados.0.smithi059.stdout:222: finishing write tid 2 to smithi05930959-41 2024-04-19T02:03:39.231 INFO:tasks.rados.rados.0.smithi059.stdout:222: finishing write tid 3 to smithi05930959-41 2024-04-19T02:03:39.231 INFO:tasks.rados.rados.0.smithi059.stdout:222: finishing write tid 4 to smithi05930959-41 2024-04-19T02:03:39.232 INFO:tasks.rados.rados.0.smithi059.stdout:222: finishing write tid 5 to smithi05930959-41 2024-04-19T02:03:39.232 INFO:tasks.rados.rados.0.smithi059.stdout:222: oid 41 updating version 0 to 59 2024-04-19T02:03:39.232 INFO:tasks.rados.rados.0.smithi059.stdout:222: oid 41 updating version 59 to 60 2024-04-19T02:03:39.232 INFO:tasks.rados.rados.0.smithi059.stdout:222: oid 41 version 60 is already newer than 58 2024-04-19T02:03:39.232 INFO:tasks.rados.rados.0.smithi059.stdout:222: oid 41 version 60 is already newer than 57 2024-04-19T02:03:39.232 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 41 v 60 (ObjNum 105 snap 14 seq_num 105) dirty exists 2024-04-19T02:03:39.232 INFO:tasks.rados.rados.0.smithi059.stdout:222: left oid 41 (ObjNum 105 snap 14 seq_num 105) 2024-04-19T02:03:39.232 INFO:tasks.rados.rados.0.smithi059.stdout:214: done (6 left) 2024-04-19T02:03:39.232 INFO:tasks.rados.rados.0.smithi059.stdout:215: done (5 left) 2024-04-19T02:03:39.232 INFO:tasks.rados.rados.0.smithi059.stdout:217: done (4 left) 2024-04-19T02:03:39.232 INFO:tasks.rados.rados.0.smithi059.stdout:218: done (3 left) 2024-04-19T02:03:39.232 INFO:tasks.rados.rados.0.smithi059.stdout:220: done (2 left) 2024-04-19T02:03:39.232 INFO:tasks.rados.rados.0.smithi059.stdout:221: done (1 left) 2024-04-19T02:03:39.232 INFO:tasks.rados.rados.0.smithi059.stdout:222: done (0 left) 2024-04-19T02:03:39.233 INFO:tasks.rados.rados.0.smithi059.stdout:223: delete oid 12 current snap is 14 2024-04-19T02:03:39.234 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-04-19T02:03:39.234 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T02:03:39.236 INFO:tasks.rados.rados.0.smithi059.stdout:223: done (0 left) 2024-04-19T02:03:39.236 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 23 current snap is 14 2024-04-19T02:03:39.236 INFO:tasks.rados.rados.0.smithi059.stdout:224: seq_num 107 ranges {0=638976,638976=458752,1097728=704512} 2024-04-19T02:03:39.236 INFO:tasks.rados.rados.0.smithi059.stdout:224: writing smithi05930959-23 from 0 to 638976 tid 1 2024-04-19T02:03:39.236 INFO:tasks.rados.rados.0.smithi059.stdout:224: writing smithi05930959-23 from 638976 to 1097728 tid 2 2024-04-19T02:03:39.236 INFO:tasks.rados.rados.0.smithi059.stdout:224: writing smithi05930959-23 from 1097728 to 1802240 tid 3 2024-04-19T02:03:39.236 INFO:tasks.rados.rados.0.smithi059.stdout:225: snap_create 2024-04-19T02:03:39.236 INFO:tasks.rados.rados.0.smithi059.stdout:224: finishing write tid 1 to smithi05930959-23 2024-04-19T02:03:39.237 INFO:tasks.rados.rados.0.smithi059.stdout:224: finishing write tid 2 to smithi05930959-23 2024-04-19T02:03:39.237 INFO:tasks.rados.rados.0.smithi059.stdout:224: finishing write tid 3 to smithi05930959-23 2024-04-19T02:03:39.237 INFO:tasks.rados.rados.0.smithi059.stdout:224: finishing write tid 4 to smithi05930959-23 2024-04-19T02:03:39.237 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-04-19T02:03:39.237 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T02:03:39.238 INFO:tasks.rados.rados.0.smithi059.stdout:224: finishing write tid 5 to smithi05930959-23 2024-04-19T02:03:39.238 INFO:tasks.rados.rados.0.smithi059.stdout:224: oid 23 updating version 0 to 111 2024-04-19T02:03:39.238 INFO:tasks.rados.rados.0.smithi059.stdout:224: oid 23 updating version 111 to 113 2024-04-19T02:03:39.238 INFO:tasks.rados.rados.0.smithi059.stdout:224: oid 23 updating version 113 to 114 2024-04-19T02:03:39.238 INFO:tasks.rados.rados.0.smithi059.stdout:224: oid 23 version 114 is already newer than 112 2024-04-19T02:03:39.238 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 23 v 114 (ObjNum 107 snap 14 seq_num 107) dirty exists 2024-04-19T02:03:39.238 INFO:tasks.rados.rados.0.smithi059.stdout:224: left oid 23 (ObjNum 107 snap 14 seq_num 107) 2024-04-19T02:03:39.238 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-04-19T02:03:39.239 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T02:03:39.239 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-04-19T02:03:39.239 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T02:03:39.239 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-04-19T02:03:39.240 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T02:03:39.240 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:03:39.234+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:03:39.255 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-04-19T02:03:39.255 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T02:03:39.275 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-04-19T02:03:39.275 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T02:03:39.337 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:39.334+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:39.364 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:39.365 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:39.365 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 0 2024-04-19T02:03:39.365 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:39.378 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-19T02:03:39.438 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:03:39.434+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:03:39.500 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:39.500 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:39.500 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-04-19T02:03:39.500 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-04-19T02:03:39.500 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:39.515 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-19T02:03:39.559 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:39.558+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:39.640 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:39.642+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:39.685 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:39.685 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-04-19T02:03:39.685 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-04-19T02:03:39.685 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-04-19T02:03:39.685 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.685 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.es1 35/26 ECSubWrite(tid=23, reqid=osd.7.0:23, at_version=35'18, trim_to=0'0, roll_forward_to=34'16))", 2024-04-19T02:03:39.686 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:19.676794+0000", 2024-04-19T02:03:39.686 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 20.009878370999999, 2024-04-19T02:03:39.686 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012162914, 2024-04-19T02:03:39.686 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:39.686 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:39.686 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:39.686 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:39.686 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.686 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:39.686 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.676794+0000", 2024-04-19T02:03:39.686 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.686 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.686 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.687 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:39.687 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.676794+0000", 2024-04-19T02:03:39.687 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.687 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.687 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.687 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:39.687 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.676795+0000", 2024-04-19T02:03:39.687 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.04e-06 2024-04-19T02:03:39.687 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.687 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.687 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:39.687 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.677097+0000", 2024-04-19T02:03:39.687 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00030220099999999999 2024-04-19T02:03:39.687 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.687 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.688 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:39.688 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.677137+0000", 2024-04-19T02:03:39.688 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9756999999999999e-05 2024-04-19T02:03:39.688 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.688 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.688 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.688 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.677442+0000", 2024-04-19T02:03:39.688 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00030463799999999998 2024-04-19T02:03:39.688 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.688 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.688 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:39.688 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.677615+0000", 2024-04-19T02:03:39.688 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017371899999999999 2024-04-19T02:03:39.688 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.689 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.689 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:39.689 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.677641+0000", 2024-04-19T02:03:39.689 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5908e-05 2024-04-19T02:03:39.689 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.689 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.689 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:39.689 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.688872+0000", 2024-04-19T02:03:39.689 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011230957999999999 2024-04-19T02:03:39.689 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.689 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.689 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:39.689 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.688957+0000", 2024-04-19T02:03:39.689 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4692999999999997e-05 2024-04-19T02:03:39.689 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.690 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:39.690 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.690 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.690 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.690 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(osd.3.18:2 3.9s0 3:956ba687:::smithi05930959-13:head [assert-version v11,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:39.690 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:24.751276+0000", 2024-04-19T02:03:39.690 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 14.935396247, 2024-04-19T02:03:39.690 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.01074665, 2024-04-19T02:03:39.690 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:39.690 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:39.690 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:03:39.690 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:39.690 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.690 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:39.690 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.751276+0000", 2024-04-19T02:03:39.691 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.691 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.691 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.691 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:39.691 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.751276+0000", 2024-04-19T02:03:39.691 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.691 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.691 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.691 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:39.691 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.751279+0000", 2024-04-19T02:03:39.691 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3499999999999999e-06 2024-04-19T02:03:39.691 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.691 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.691 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:39.691 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.751311+0000", 2024-04-19T02:03:39.692 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.2808000000000002e-05 2024-04-19T02:03:39.692 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.692 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.692 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:39.692 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.751344+0000", 2024-04-19T02:03:39.692 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3124000000000003e-05 2024-04-19T02:03:39.692 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.692 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.692 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.692 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.751367+0000", 2024-04-19T02:03:39.692 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2768e-05 2024-04-19T02:03:39.692 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.692 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.692 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:39.692 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.751590+0000", 2024-04-19T02:03:39.693 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00022258099999999999 2024-04-19T02:03:39.693 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.693 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.693 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op must wait for map", 2024-04-19T02:03:39.693 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.751646+0000", 2024-04-19T02:03:39.693 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6586000000000003e-05 2024-04-19T02:03:39.693 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.693 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.693 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:39.693 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.756977+0000", 2024-04-19T02:03:39.693 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0053301069999999997 2024-04-19T02:03:39.693 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.693 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.693 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:39.693 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.757235+0000", 2024-04-19T02:03:39.693 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00025816599999999999 2024-04-19T02:03:39.694 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.694 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.694 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:39.694 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.762023+0000", 2024-04-19T02:03:39.694 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0047881599999999996 2024-04-19T02:03:39.694 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.694 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:39.694 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.694 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.694 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.694 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.2s3 42/26 ECSubWrite(tid=132, reqid=osd.3.0:132, at_version=42'63, trim_to=0'0, roll_forward_to=40'61))", 2024-04-19T02:03:39.694 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.747807+0000", 2024-04-19T02:03:39.694 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 12.938865677000001, 2024-04-19T02:03:39.694 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010410005, 2024-04-19T02:03:39.694 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:39.695 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:39.695 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:39.695 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:39.695 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.695 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:39.695 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.747807+0000", 2024-04-19T02:03:39.695 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.695 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.695 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.695 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:39.695 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.747807+0000", 2024-04-19T02:03:39.695 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.695 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.695 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.695 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:39.696 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.747808+0000", 2024-04-19T02:03:39.696 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.717e-06 2024-04-19T02:03:39.696 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.696 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.696 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:39.696 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.747876+0000", 2024-04-19T02:03:39.696 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7582000000000004e-05 2024-04-19T02:03:39.696 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.696 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.696 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:39.696 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.747918+0000", 2024-04-19T02:03:39.696 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2426999999999999e-05 2024-04-19T02:03:39.696 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.696 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.696 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.697 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.747941+0000", 2024-04-19T02:03:39.697 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2974999999999999e-05 2024-04-19T02:03:39.697 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.697 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.697 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:39.697 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748091+0000", 2024-04-19T02:03:39.697 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000149844 2024-04-19T02:03:39.697 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.697 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.697 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:39.697 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748118+0000", 2024-04-19T02:03:39.697 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6754000000000002e-05 2024-04-19T02:03:39.697 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.697 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.697 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:39.698 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.758136+0000", 2024-04-19T02:03:39.698 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010017814999999999 2024-04-19T02:03:39.698 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.698 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.698 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:39.698 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.758217+0000", 2024-04-19T02:03:39.698 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0890999999999995e-05 2024-04-19T02:03:39.698 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.698 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:39.698 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.698 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.698 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.698 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.cs4 46/26 ECSubWrite(tid=44, reqid=osd.9.0:44, at_version=46'26, trim_to=0'0, roll_forward_to=45'24))", 2024-04-19T02:03:39.698 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.065744+0000", 2024-04-19T02:03:39.699 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 9.6209283540000001, 2024-04-19T02:03:39.699 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.018889659, 2024-04-19T02:03:39.699 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:39.699 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:39.699 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:39.699 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:39.699 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.699 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:39.699 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.065744+0000", 2024-04-19T02:03:39.699 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.699 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.699 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.699 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:39.699 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.065744+0000", 2024-04-19T02:03:39.699 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.700 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.700 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.700 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:39.700 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.065745+0000", 2024-04-19T02:03:39.700 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0999999999999998e-07 2024-04-19T02:03:39.700 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.700 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.700 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:39.700 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.065794+0000", 2024-04-19T02:03:39.700 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9177e-05 2024-04-19T02:03:39.700 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.700 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.700 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:39.700 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.065836+0000", 2024-04-19T02:03:39.700 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2245e-05 2024-04-19T02:03:39.701 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.701 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.701 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.701 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.065860+0000", 2024-04-19T02:03:39.701 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3654000000000001e-05 2024-04-19T02:03:39.701 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.701 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.701 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:39.701 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066138+0000", 2024-04-19T02:03:39.701 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00027793700000000002 2024-04-19T02:03:39.701 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.701 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.701 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:39.701 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066164+0000", 2024-04-19T02:03:39.701 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5862000000000001e-05 2024-04-19T02:03:39.702 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.702 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.702 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:39.702 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.084533+0000", 2024-04-19T02:03:39.702 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.018369534 2024-04-19T02:03:39.702 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.702 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.702 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:39.702 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.084634+0000", 2024-04-19T02:03:39.702 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010054 2024-04-19T02:03:39.702 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.702 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:39.702 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.702 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.702 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.703 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:431 3.8s0 3:1984c4bc:::smithi05930959-40:head [read 0~3162112,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:39.703 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.193642+0000", 2024-04-19T02:03:39.703 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 9.4930300009999993, 2024-04-19T02:03:39.703 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.015491476000000001, 2024-04-19T02:03:39.703 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:39.703 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:39.703 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:03:39.703 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:39.703 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:39.703 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:39.703 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 431 2024-04-19T02:03:39.703 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.703 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:39.703 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.704 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:39.704 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193642+0000", 2024-04-19T02:03:39.704 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.704 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.704 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.704 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:39.704 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193642+0000", 2024-04-19T02:03:39.704 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.704 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.704 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.704 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:39.704 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193645+0000", 2024-04-19T02:03:39.704 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0369999999999999e-06 2024-04-19T02:03:39.704 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.704 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.705 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:39.705 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193659+0000", 2024-04-19T02:03:39.705 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3424999999999999e-05 2024-04-19T02:03:39.705 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.705 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.705 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:39.705 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193692+0000", 2024-04-19T02:03:39.705 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3382e-05 2024-04-19T02:03:39.705 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.705 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.705 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.705 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193714+0000", 2024-04-19T02:03:39.705 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1522000000000002e-05 2024-04-19T02:03:39.705 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.706 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.706 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:39.706 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193838+0000", 2024-04-19T02:03:39.706 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012381499999999999 2024-04-19T02:03:39.706 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.706 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.706 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:39.706 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.194001+0000", 2024-04-19T02:03:39.706 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016321 2024-04-19T02:03:39.706 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.706 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.706 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:39.706 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.209134+0000", 2024-04-19T02:03:39.706 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.015133084999999999 2024-04-19T02:03:39.707 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.707 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:39.707 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.707 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.707 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.707 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:432 3.8s0 3:1984c4bc:::smithi05930959-40:head [read 0~3162112,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:39.707 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.193759+0000", 2024-04-19T02:03:39.707 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 9.4929131709999997, 2024-04-19T02:03:39.707 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.020892661, 2024-04-19T02:03:39.707 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:39.707 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:39.707 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:03:39.707 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:39.707 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:39.707 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:39.708 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 432 2024-04-19T02:03:39.708 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.708 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:39.708 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.708 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:39.708 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193759+0000", 2024-04-19T02:03:39.708 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.708 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.708 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.708 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:39.708 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193759+0000", 2024-04-19T02:03:39.708 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.708 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.708 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.708 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:39.708 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193761+0000", 2024-04-19T02:03:39.709 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3430000000000001e-06 2024-04-19T02:03:39.709 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.709 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.709 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:39.709 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193770+0000", 2024-04-19T02:03:39.709 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5359999999999998e-06 2024-04-19T02:03:39.709 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.709 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.709 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:39.709 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193794+0000", 2024-04-19T02:03:39.709 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3589000000000001e-05 2024-04-19T02:03:39.709 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.709 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.709 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.709 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193811+0000", 2024-04-19T02:03:39.710 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6838999999999999e-05 2024-04-19T02:03:39.710 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.710 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.710 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:39.710 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.194657+0000", 2024-04-19T02:03:39.710 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00084614800000000002 2024-04-19T02:03:39.710 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.710 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.710 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:39.710 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.194925+0000", 2024-04-19T02:03:39.710 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000267927 2024-04-19T02:03:39.710 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.710 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.710 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:39.711 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.214652+0000", 2024-04-19T02:03:39.711 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.019727279 2024-04-19T02:03:39.711 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.711 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:39.711 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.711 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.711 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.711 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:433 3.8s0 3:1984c4bc:::smithi05930959-40:head [read 0~3162112,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:39.711 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.193839+0000", 2024-04-19T02:03:39.711 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 9.4928337529999993, 2024-04-19T02:03:39.711 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.025594295, 2024-04-19T02:03:39.711 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:39.711 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:39.711 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:03:39.711 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:39.711 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:39.712 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:39.712 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 433 2024-04-19T02:03:39.712 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.712 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:39.712 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.712 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:39.712 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193839+0000", 2024-04-19T02:03:39.712 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.712 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.712 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.712 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:39.712 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193839+0000", 2024-04-19T02:03:39.712 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.713 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.713 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.713 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:39.713 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193840+0000", 2024-04-19T02:03:39.713 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.277e-06 2024-04-19T02:03:39.713 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.713 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.713 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:39.713 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193848+0000", 2024-04-19T02:03:39.713 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0790000000000001e-06 2024-04-19T02:03:39.713 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.713 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.713 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:39.713 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193866+0000", 2024-04-19T02:03:39.713 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8411999999999999e-05 2024-04-19T02:03:39.713 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.713 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.714 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.714 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193881+0000", 2024-04-19T02:03:39.714 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4419e-05 2024-04-19T02:03:39.714 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.714 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.714 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:39.714 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.202091+0000", 2024-04-19T02:03:39.714 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0082103079999999995 2024-04-19T02:03:39.714 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.714 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.714 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:39.714 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.202540+0000", 2024-04-19T02:03:39.714 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00044865600000000002 2024-04-19T02:03:39.714 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.715 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.715 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:39.715 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.219433+0000", 2024-04-19T02:03:39.715 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.016893143999999999 2024-04-19T02:03:39.715 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.715 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:39.715 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.715 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.715 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.715 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpReadReply(3.8s0 46/26 ECSubReadReply(tid=84, attrs_read=0))", 2024-04-19T02:03:39.715 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.198517+0000", 2024-04-19T02:03:39.715 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 9.4881551609999999, 2024-04-19T02:03:39.715 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011414418000000001, 2024-04-19T02:03:39.715 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:39.716 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:39.716 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:39.716 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:39.716 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.716 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:39.716 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.198517+0000", 2024-04-19T02:03:39.716 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.716 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.716 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.716 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:39.716 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.198517+0000", 2024-04-19T02:03:39.716 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.716 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.716 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.717 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:39.717 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.198518+0000", 2024-04-19T02:03:39.717 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.167e-06 2024-04-19T02:03:39.717 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.717 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.717 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:39.717 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.200086+0000", 2024-04-19T02:03:39.717 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0015677810000000001 2024-04-19T02:03:39.717 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.717 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.717 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:39.717 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.200115+0000", 2024-04-19T02:03:39.717 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8602000000000002e-05 2024-04-19T02:03:39.717 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.717 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.718 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.718 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.200133+0000", 2024-04-19T02:03:39.718 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7955e-05 2024-04-19T02:03:39.718 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.718 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.718 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:39.718 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.209825+0000", 2024-04-19T02:03:39.718 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0096922050000000006 2024-04-19T02:03:39.718 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.718 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.718 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:39.718 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.209932+0000", 2024-04-19T02:03:39.718 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000106708 2024-04-19T02:03:39.718 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.719 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:39.719 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.719 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.719 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.719 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.ds3 46/26 ECSubWrite(tid=190, reqid=client.4509.0:428, at_version=46'37, trim_to=0'0, roll_forward_to=46'35))", 2024-04-19T02:03:39.719 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.200233+0000", 2024-04-19T02:03:39.719 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 9.4864390889999992, 2024-04-19T02:03:39.719 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010604951, 2024-04-19T02:03:39.719 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:39.719 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:39.719 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:39.719 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:39.719 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.719 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:39.719 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.200233+0000", 2024-04-19T02:03:39.720 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.720 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.720 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.720 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:39.720 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.200233+0000", 2024-04-19T02:03:39.720 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.720 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.720 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.720 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:39.720 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.200235+0000", 2024-04-19T02:03:39.720 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.39e-06 2024-04-19T02:03:39.720 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.720 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.720 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:39.720 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.200279+0000", 2024-04-19T02:03:39.721 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4190999999999998e-05 2024-04-19T02:03:39.721 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.721 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.721 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:39.721 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.200305+0000", 2024-04-19T02:03:39.721 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6449e-05 2024-04-19T02:03:39.721 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.721 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.721 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.721 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.200325+0000", 2024-04-19T02:03:39.721 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9324999999999999e-05 2024-04-19T02:03:39.721 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.721 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.721 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:39.722 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.201816+0000", 2024-04-19T02:03:39.722 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0014913859999999999 2024-04-19T02:03:39.722 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.722 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.722 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:39.722 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.201842+0000", 2024-04-19T02:03:39.722 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6024999999999999e-05 2024-04-19T02:03:39.722 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.722 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.722 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:39.722 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.210719+0000", 2024-04-19T02:03:39.722 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0088765619999999993 2024-04-19T02:03:39.722 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.722 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.723 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:39.723 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.210838+0000", 2024-04-19T02:03:39.723 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000119623 2024-04-19T02:03:39.723 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.723 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:39.723 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.723 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.723 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.723 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.2s3 46/26 ECSubWrite(tid=163, reqid=client.4509.0:425, at_version=46'78, trim_to=0'0, roll_forward_to=46'75))", 2024-04-19T02:03:39.723 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.200530+0000", 2024-04-19T02:03:39.723 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 9.4861425930000003, 2024-04-19T02:03:39.723 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010325826999999999, 2024-04-19T02:03:39.723 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:39.723 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:39.723 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:39.723 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:39.724 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.724 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:39.724 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.200530+0000", 2024-04-19T02:03:39.724 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.724 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.724 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.724 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:39.724 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.200530+0000", 2024-04-19T02:03:39.724 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.724 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.724 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.724 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:39.724 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.200531+0000", 2024-04-19T02:03:39.724 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2359999999999999e-06 2024-04-19T02:03:39.724 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.725 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.725 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:39.725 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.200572+0000", 2024-04-19T02:03:39.725 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0935e-05 2024-04-19T02:03:39.725 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.725 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.725 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:39.725 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.200599+0000", 2024-04-19T02:03:39.725 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7418000000000001e-05 2024-04-19T02:03:39.725 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.725 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.725 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.725 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.200618+0000", 2024-04-19T02:03:39.725 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8156e-05 2024-04-19T02:03:39.725 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.726 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.726 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:39.726 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.200776+0000", 2024-04-19T02:03:39.726 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015842699999999999 2024-04-19T02:03:39.726 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.726 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.726 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:39.726 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.200802+0000", 2024-04-19T02:03:39.726 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6322e-05 2024-04-19T02:03:39.726 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.726 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.726 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:39.726 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.210774+0000", 2024-04-19T02:03:39.726 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0099718510000000003 2024-04-19T02:03:39.727 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.727 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.727 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:39.727 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.210856+0000", 2024-04-19T02:03:39.727 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.1482000000000004e-05 2024-04-19T02:03:39.727 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.727 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:39.727 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.727 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.727 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.727 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.bs4 47/26 ECSubWrite(tid=180, reqid=osd.3.0:180, at_version=47'36, trim_to=0'0, roll_forward_to=46'34))", 2024-04-19T02:03:39.727 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.080143+0000", 2024-04-19T02:03:39.727 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 8.6065296080000007, 2024-04-19T02:03:39.727 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012272063, 2024-04-19T02:03:39.727 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:39.728 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:39.728 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:39.728 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:39.728 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.728 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:39.728 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.080143+0000", 2024-04-19T02:03:39.728 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.728 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.728 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.728 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:39.728 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.080143+0000", 2024-04-19T02:03:39.728 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.728 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.728 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.728 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:39.729 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.080144+0000", 2024-04-19T02:03:39.729 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.266e-06 2024-04-19T02:03:39.729 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.729 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.729 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:39.729 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.080191+0000", 2024-04-19T02:03:39.729 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6693999999999997e-05 2024-04-19T02:03:39.729 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.729 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.729 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:39.729 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081012+0000", 2024-04-19T02:03:39.729 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00082073400000000005 2024-04-19T02:03:39.729 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.729 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.729 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.729 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081033+0000", 2024-04-19T02:03:39.730 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1648999999999999e-05 2024-04-19T02:03:39.730 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.730 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.730 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:39.730 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081209+0000", 2024-04-19T02:03:39.730 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017594 2024-04-19T02:03:39.730 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.730 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.730 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:39.730 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081243+0000", 2024-04-19T02:03:39.730 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3976999999999997e-05 2024-04-19T02:03:39.730 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.730 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.730 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:39.731 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.092330+0000", 2024-04-19T02:03:39.731 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011086972 2024-04-19T02:03:39.731 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.731 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.731 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:39.731 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.092415+0000", 2024-04-19T02:03:39.731 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4831000000000001e-05 2024-04-19T02:03:39.731 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.731 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:39.731 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.731 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.731 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.731 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.2s3 47/26 ECSubWrite(tid=182, reqid=osd.3.0:182, at_version=47'89, trim_to=0'0, roll_forward_to=46'85))", 2024-04-19T02:03:39.731 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.081368+0000", 2024-04-19T02:03:39.731 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 8.6053046949999992, 2024-04-19T02:03:39.732 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011123199, 2024-04-19T02:03:39.732 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:39.732 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:39.732 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:39.732 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:39.732 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.732 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:39.732 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081368+0000", 2024-04-19T02:03:39.732 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.732 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.732 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.732 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:39.732 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081368+0000", 2024-04-19T02:03:39.732 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.732 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.732 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.733 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:39.733 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081370+0000", 2024-04-19T02:03:39.733 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.854e-06 2024-04-19T02:03:39.733 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.733 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.733 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:39.733 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081429+0000", 2024-04-19T02:03:39.733 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9289000000000001e-05 2024-04-19T02:03:39.733 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.733 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.733 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:39.733 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081463+0000", 2024-04-19T02:03:39.733 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3977999999999999e-05 2024-04-19T02:03:39.733 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.733 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.734 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.734 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081485+0000", 2024-04-19T02:03:39.734 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1999000000000001e-05 2024-04-19T02:03:39.734 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.734 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.734 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:39.734 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081652+0000", 2024-04-19T02:03:39.734 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016734000000000001 2024-04-19T02:03:39.734 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.734 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.734 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:39.734 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081676+0000", 2024-04-19T02:03:39.734 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3915000000000001e-05 2024-04-19T02:03:39.734 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.734 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.735 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:39.735 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.092407+0000", 2024-04-19T02:03:39.735 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010731042999999999 2024-04-19T02:03:39.735 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.735 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.735 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:39.735 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.092491+0000", 2024-04-19T02:03:39.735 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3781000000000003e-05 2024-04-19T02:03:39.735 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.735 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:39.735 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.735 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.735 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.735 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.bs4 47/26 ECSubWrite(tid=183, reqid=osd.3.0:183, at_version=47'38, trim_to=0'0, roll_forward_to=46'34))", 2024-04-19T02:03:39.735 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.081558+0000", 2024-04-19T02:03:39.736 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 8.6051140860000004, 2024-04-19T02:03:39.736 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014283580000000001, 2024-04-19T02:03:39.736 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:39.736 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:39.736 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:39.736 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:39.736 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.736 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:39.736 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081558+0000", 2024-04-19T02:03:39.736 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.736 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.736 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.736 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:39.736 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081558+0000", 2024-04-19T02:03:39.736 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.736 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.737 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.737 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:39.737 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081559+0000", 2024-04-19T02:03:39.737 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4300000000000001e-07 2024-04-19T02:03:39.737 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.737 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.737 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:39.737 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081613+0000", 2024-04-19T02:03:39.737 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3560000000000002e-05 2024-04-19T02:03:39.737 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.737 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.737 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:39.737 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081636+0000", 2024-04-19T02:03:39.737 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3266000000000001e-05 2024-04-19T02:03:39.737 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.737 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.738 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.738 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081653+0000", 2024-04-19T02:03:39.738 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6821999999999999e-05 2024-04-19T02:03:39.738 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.738 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.738 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:39.738 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.082251+0000", 2024-04-19T02:03:39.738 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00059773699999999999 2024-04-19T02:03:39.738 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.738 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.738 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:39.738 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.082276+0000", 2024-04-19T02:03:39.738 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5066e-05 2024-04-19T02:03:39.738 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.739 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.739 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:39.739 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.095735+0000", 2024-04-19T02:03:39.739 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013459575999999999 2024-04-19T02:03:39.739 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.739 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.739 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:39.739 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.095842+0000", 2024-04-19T02:03:39.739 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010661 2024-04-19T02:03:39.739 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.739 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:39.739 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.739 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.739 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.739 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.ds3 47/26 ECSubWrite(tid=193, reqid=osd.8.0:193, at_version=47'40, trim_to=0'0, roll_forward_to=46'38))", 2024-04-19T02:03:39.740 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.083005+0000", 2024-04-19T02:03:39.740 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 8.6036677390000005, 2024-04-19T02:03:39.740 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013056883, 2024-04-19T02:03:39.740 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:39.740 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:39.740 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:39.740 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:39.740 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.740 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:39.740 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.083005+0000", 2024-04-19T02:03:39.740 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.740 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.740 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.740 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:39.741 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.083005+0000", 2024-04-19T02:03:39.741 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.741 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.741 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.741 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:39.741 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.083005+0000", 2024-04-19T02:03:39.741 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9299999999999998e-07 2024-04-19T02:03:39.741 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.741 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.741 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:39.741 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.083052+0000", 2024-04-19T02:03:39.741 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7036999999999997e-05 2024-04-19T02:03:39.741 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.741 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.741 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:39.742 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.083082+0000", 2024-04-19T02:03:39.742 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9451999999999999e-05 2024-04-19T02:03:39.742 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.742 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.742 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.742 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.083110+0000", 2024-04-19T02:03:39.742 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8317999999999999e-05 2024-04-19T02:03:39.742 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.742 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.742 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:39.742 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.083268+0000", 2024-04-19T02:03:39.742 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015767099999999999 2024-04-19T02:03:39.742 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.742 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.743 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:39.743 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.083293+0000", 2024-04-19T02:03:39.743 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5408000000000001e-05 2024-04-19T02:03:39.743 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.743 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.743 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:39.743 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.095974+0000", 2024-04-19T02:03:39.743 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012681289 2024-04-19T02:03:39.743 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.743 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.743 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:39.743 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.096062+0000", 2024-04-19T02:03:39.743 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7114999999999995e-05 2024-04-19T02:03:39.743 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.743 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:39.743 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.744 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.744 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.744 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.1s2 47/26 ECSubWrite(tid=30, reqid=osd.0.0:30, at_version=47'17, trim_to=0'0, roll_forward_to=46'13))", 2024-04-19T02:03:39.744 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.086105+0000", 2024-04-19T02:03:39.744 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 8.6005672779999998, 2024-04-19T02:03:39.744 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013306735, 2024-04-19T02:03:39.744 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:39.744 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:39.744 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:39.744 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:39.744 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.744 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:39.744 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.086105+0000", 2024-04-19T02:03:39.744 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.744 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.745 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.745 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:39.745 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.086105+0000", 2024-04-19T02:03:39.745 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.745 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.745 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.745 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:39.745 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.086106+0000", 2024-04-19T02:03:39.745 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.2699999999999998e-07 2024-04-19T02:03:39.745 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.746 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.746 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:39.746 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.086151+0000", 2024-04-19T02:03:39.746 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5256999999999997e-05 2024-04-19T02:03:39.746 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.746 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.746 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:39.746 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.086175+0000", 2024-04-19T02:03:39.746 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3805999999999999e-05 2024-04-19T02:03:39.747 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.747 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.747 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.747 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.086192+0000", 2024-04-19T02:03:39.747 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7175000000000001e-05 2024-04-19T02:03:39.747 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.747 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.747 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:39.747 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.089760+0000", 2024-04-19T02:03:39.747 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0035675569999999998 2024-04-19T02:03:39.747 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.747 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.748 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:39.748 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.089783+0000", 2024-04-19T02:03:39.748 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3575e-05 2024-04-19T02:03:39.748 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.748 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.748 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:39.748 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.099331+0000", 2024-04-19T02:03:39.748 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0095480470000000005 2024-04-19T02:03:39.748 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.748 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.749 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:39.749 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.099412+0000", 2024-04-19T02:03:39.749 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0390999999999997e-05 2024-04-19T02:03:39.749 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.749 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:39.749 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.749 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.749 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.749 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:470 3.9s0 3:97177082:::smithi05930959-45:head [append 0~802816] snapc a=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:03:39.749 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.167632+0000", 2024-04-19T02:03:39.749 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 8.5190402600000006, 2024-04-19T02:03:39.750 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011924591999999999, 2024-04-19T02:03:39.750 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:39.750 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:39.750 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:39.750 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:39.750 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:39.750 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:39.750 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 470 2024-04-19T02:03:39.750 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.750 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:39.750 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.751 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:39.751 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.167632+0000", 2024-04-19T02:03:39.751 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.751 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.751 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.751 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:39.751 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.167632+0000", 2024-04-19T02:03:39.751 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.751 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.751 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.751 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:39.751 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.167636+0000", 2024-04-19T02:03:39.752 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5200000000000002e-06 2024-04-19T02:03:39.752 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.752 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.752 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:39.752 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.169434+0000", 2024-04-19T02:03:39.752 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0017987890000000001 2024-04-19T02:03:39.752 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.752 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.752 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:39.752 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.169461+0000", 2024-04-19T02:03:39.752 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6035999999999999e-05 2024-04-19T02:03:39.752 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.753 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.753 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.753 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.169479+0000", 2024-04-19T02:03:39.753 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8241999999999999e-05 2024-04-19T02:03:39.753 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.753 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.753 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:39.753 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.169620+0000", 2024-04-19T02:03:39.753 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014108100000000001 2024-04-19T02:03:39.753 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.753 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.753 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:39.754 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.169850+0000", 2024-04-19T02:03:39.754 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000229702 2024-04-19T02:03:39.754 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.754 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.754 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:39.754 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.173203+0000", 2024-04-19T02:03:39.754 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0033530970000000002 2024-04-19T02:03:39.754 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.754 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.754 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:39.754 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.178706+0000", 2024-04-19T02:03:39.755 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0055031979999999999 2024-04-19T02:03:39.755 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.755 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.755 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:39.755 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.179377+0000", 2024-04-19T02:03:39.755 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00067095999999999996 2024-04-19T02:03:39.755 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.755 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.755 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:39.755 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.179557+0000", 2024-04-19T02:03:39.755 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017996700000000001 2024-04-19T02:03:39.755 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.755 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:39.755 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.755 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.756 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.756 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:471 3.9s0 3:97177082:::smithi05930959-45:head [append 0~638976] snapc a=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:03:39.756 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.171946+0000", 2024-04-19T02:03:39.756 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 8.5147268409999999, 2024-04-19T02:03:39.756 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010415407999999999, 2024-04-19T02:03:39.756 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:39.756 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:39.756 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:39.756 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:39.756 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:39.756 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:39.756 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 471 2024-04-19T02:03:39.757 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.757 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:39.757 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.757 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:39.757 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.171946+0000", 2024-04-19T02:03:39.757 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.757 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.757 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.757 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:39.757 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.171946+0000", 2024-04-19T02:03:39.757 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.757 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.758 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.758 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:39.758 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.171948+0000", 2024-04-19T02:03:39.758 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.306e-06 2024-04-19T02:03:39.758 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.758 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.758 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:39.758 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.173637+0000", 2024-04-19T02:03:39.758 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001689126 2024-04-19T02:03:39.758 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.758 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.759 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:39.759 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.173668+0000", 2024-04-19T02:03:39.759 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1145000000000001e-05 2024-04-19T02:03:39.759 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.759 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.759 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.759 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.173691+0000", 2024-04-19T02:03:39.759 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2892e-05 2024-04-19T02:03:39.759 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.759 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.759 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:39.759 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.175087+0000", 2024-04-19T02:03:39.759 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0013960489999999999 2024-04-19T02:03:39.760 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.760 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.760 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:39.760 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.175229+0000", 2024-04-19T02:03:39.760 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014216899999999999 2024-04-19T02:03:39.760 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.760 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.760 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:39.760 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.177004+0000", 2024-04-19T02:03:39.760 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0017746140000000001 2024-04-19T02:03:39.760 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.760 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.760 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:39.761 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.179578+0000", 2024-04-19T02:03:39.761 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0025745020000000002 2024-04-19T02:03:39.761 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.761 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.761 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:39.761 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.182202+0000", 2024-04-19T02:03:39.761 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0026237919999999998 2024-04-19T02:03:39.761 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.761 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.761 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:39.761 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.182361+0000", 2024-04-19T02:03:39.761 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000158813 2024-04-19T02:03:39.761 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.762 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:39.762 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.762 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.762 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.762 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:472 3.9s0 3:97177082:::smithi05930959-45:head [append 0~131072] snapc a=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:03:39.762 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.173736+0000", 2024-04-19T02:03:39.762 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 8.5129367049999995, 2024-04-19T02:03:39.762 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011890121999999999, 2024-04-19T02:03:39.762 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:39.762 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:39.762 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:39.762 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:39.762 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:39.763 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:39.763 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 472 2024-04-19T02:03:39.763 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.763 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:39.763 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.763 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:39.763 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.173736+0000", 2024-04-19T02:03:39.763 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.763 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.763 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.763 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:39.763 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.173736+0000", 2024-04-19T02:03:39.764 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.764 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.764 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.764 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:39.764 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.173738+0000", 2024-04-19T02:03:39.764 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.86e-06 2024-04-19T02:03:39.764 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.764 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.764 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:39.764 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.174012+0000", 2024-04-19T02:03:39.764 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000274293 2024-04-19T02:03:39.764 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.764 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.764 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:39.765 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.174039+0000", 2024-04-19T02:03:39.765 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7396e-05 2024-04-19T02:03:39.765 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.765 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.765 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.765 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.174059+0000", 2024-04-19T02:03:39.765 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9442000000000001e-05 2024-04-19T02:03:39.765 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.765 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.765 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:39.765 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.180105+0000", 2024-04-19T02:03:39.765 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0060463849999999996 2024-04-19T02:03:39.766 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.766 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.766 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:39.766 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.180245+0000", 2024-04-19T02:03:39.766 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000140371 2024-04-19T02:03:39.766 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.766 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.766 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:39.766 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.181270+0000", 2024-04-19T02:03:39.766 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001024713 2024-04-19T02:03:39.766 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.766 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.766 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:39.767 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.184920+0000", 2024-04-19T02:03:39.767 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0036498239999999999 2024-04-19T02:03:39.767 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.767 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.767 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:39.767 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.185501+0000", 2024-04-19T02:03:39.767 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00058073399999999996 2024-04-19T02:03:39.767 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.767 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.767 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:39.767 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.185626+0000", 2024-04-19T02:03:39.768 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012510399999999999 2024-04-19T02:03:39.768 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.768 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:39.768 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.768 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.768 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.768 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:473 3.9s0 3:97177082:::smithi05930959-45:head [setxattr _header (56) in=7b] snapc a=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:03:39.768 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.174085+0000", 2024-04-19T02:03:39.768 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 8.5125871130000004, 2024-04-19T02:03:39.768 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012468714000000001, 2024-04-19T02:03:39.768 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:39.769 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:39.769 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:39.769 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:39.769 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:39.769 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:39.769 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 473 2024-04-19T02:03:39.769 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.769 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:39.769 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.769 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:39.769 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.174085+0000", 2024-04-19T02:03:39.769 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.770 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.770 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.770 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:39.770 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.174085+0000", 2024-04-19T02:03:39.770 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.770 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.770 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.770 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:39.770 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.174087+0000", 2024-04-19T02:03:39.770 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.567e-06 2024-04-19T02:03:39.770 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.771 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.771 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:39.771 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.174096+0000", 2024-04-19T02:03:39.771 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8929999999999994e-06 2024-04-19T02:03:39.771 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.771 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.771 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:39.771 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.174115+0000", 2024-04-19T02:03:39.771 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9445000000000001e-05 2024-04-19T02:03:39.771 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.771 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.772 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.772 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.174132+0000", 2024-04-19T02:03:39.772 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6651999999999999e-05 2024-04-19T02:03:39.772 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.772 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.772 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:39.772 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.182528+0000", 2024-04-19T02:03:39.772 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0083963030000000008 2024-04-19T02:03:39.772 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.772 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.772 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:39.773 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.182781+0000", 2024-04-19T02:03:39.773 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00025269700000000002 2024-04-19T02:03:39.773 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.773 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.773 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:39.773 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.183985+0000", 2024-04-19T02:03:39.773 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0012036530000000001 2024-04-19T02:03:39.773 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.773 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.773 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:39.774 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.185647+0000", 2024-04-19T02:03:39.774 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001662941 2024-04-19T02:03:39.774 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.774 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.774 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:39.774 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.186412+0000", 2024-04-19T02:03:39.774 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00076417200000000003 2024-04-19T02:03:39.774 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.774 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.774 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:39.774 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.186554+0000", 2024-04-19T02:03:39.775 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014239100000000001 2024-04-19T02:03:39.775 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.775 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:39.775 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.775 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.775 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.775 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:474 3.9s0 3:97177082:::smithi05930959-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:03:39.775 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.174152+0000", 2024-04-19T02:03:39.775 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 8.5125206630000001, 2024-04-19T02:03:39.775 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.016567683999999999, 2024-04-19T02:03:39.775 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:39.776 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:39.776 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:03:39.776 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:39.776 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:39.776 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:39.776 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 474 2024-04-19T02:03:39.776 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.776 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:39.776 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.776 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:39.776 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.174152+0000", 2024-04-19T02:03:39.776 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.776 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.777 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.777 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:39.777 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.174152+0000", 2024-04-19T02:03:39.777 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:39.777 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.777 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.777 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:39.777 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.174153+0000", 2024-04-19T02:03:39.777 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.39e-06 2024-04-19T02:03:39.777 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.777 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.778 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:39.778 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.174159+0000", 2024-04-19T02:03:39.778 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4759999999999997e-06 2024-04-19T02:03:39.778 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.778 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.778 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:39.778 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.174179+0000", 2024-04-19T02:03:39.778 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9966000000000002e-05 2024-04-19T02:03:39.778 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.778 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.778 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:39.778 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.174194+0000", 2024-04-19T02:03:39.779 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5332000000000002e-05 2024-04-19T02:03:39.779 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.779 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.779 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:39.779 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.187276+0000", 2024-04-19T02:03:39.779 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013081603000000001 2024-04-19T02:03:39.779 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.779 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.779 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:39.779 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.187522+0000", 2024-04-19T02:03:39.779 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00024612499999999998 2024-04-19T02:03:39.779 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:39.780 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:39.780 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:39.780 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.190719+0000", 2024-04-19T02:03:39.780 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0031977920000000001 2024-04-19T02:03:39.780 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.780 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:39.780 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.780 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:39.780 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:39.780 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:39.782 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-19T02:03:39.783 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:39.782+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:39.883 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:39.882+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:39.909 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:39.909 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:39.909 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 0 2024-04-19T02:03:39.909 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:39.923 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-19T02:03:39.983 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:03:39.982+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:03:40.041 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:40.041 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:40.042 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-04-19T02:03:40.042 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-04-19T02:03:40.042 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:40.055 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-19T02:03:40.084 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:40.082+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:40.121 INFO:teuthology.orchestra.run.smithi059.stderr:set pool 3 pgp_num to 16 2024-04-19T02:03:40.127 INFO:tasks.rados.rados.0.smithi059.stdout:224: done (1 left) 2024-04-19T02:03:40.127 INFO:tasks.rados.rados.0.smithi059.stdout:225: done (0 left) 2024-04-19T02:03:40.127 INFO:tasks.rados.rados.0.smithi059.stdout:226: rmattr oid 32 current snap is 15 2024-04-19T02:03:40.127 INFO:tasks.rados.rados.0.smithi059.stdout:226: done (0 left) 2024-04-19T02:03:40.127 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 3 current snap is 15 2024-04-19T02:03:40.128 INFO:tasks.rados.rados.0.smithi059.stdout:227: seq_num 108 ranges {1802240=655360,2457600=606208,3063808=475136,3538944=81920} 2024-04-19T02:03:40.157 INFO:tasks.rados.rados.0.smithi059.stdout:227: writing smithi05930959-3 from 1802240 to 2457600 tid 1 2024-04-19T02:03:40.165 INFO:tasks.rados.rados.0.smithi059.stdout:227: writing smithi05930959-3 from 2457600 to 3063808 tid 2 2024-04-19T02:03:40.170 INFO:tasks.rados.rados.0.smithi059.stdout:227: writing smithi05930959-3 from 3063808 to 3538944 tid 3 2024-04-19T02:03:40.172 INFO:tasks.rados.rados.0.smithi059.stdout:227: writing smithi05930959-3 from 3538944 to 3620864 tid 4 2024-04-19T02:03:40.172 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 46 current snap is 15 2024-04-19T02:03:40.172 INFO:tasks.rados.rados.0.smithi059.stdout:228: seq_num 109 ranges {2408448=671744,3080192=753664,3833856=409600} 2024-04-19T02:03:40.184 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:40.182+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:40.203 INFO:tasks.rados.rados.0.smithi059.stdout:228: writing smithi05930959-46 from 2408448 to 3080192 tid 1 2024-04-19T02:03:40.209 INFO:tasks.rados.rados.0.smithi059.stdout:228: writing smithi05930959-46 from 3080192 to 3833856 tid 2 2024-04-19T02:03:40.213 INFO:tasks.rados.rados.0.smithi059.stdout:228: writing smithi05930959-46 from 3833856 to 4243456 tid 3 2024-04-19T02:03:40.213 INFO:tasks.rados.rados.0.smithi059.stdout:229: delete oid 38 current snap is 15 2024-04-19T02:03:40.213 INFO:tasks.rados.rados.0.smithi059.stdout:227: finishing write tid 1 to smithi05930959-3 2024-04-19T02:03:40.213 INFO:tasks.rados.rados.0.smithi059.stdout:227: finishing write tid 2 to smithi05930959-3 2024-04-19T02:03:40.213 INFO:tasks.rados.rados.0.smithi059.stdout:227: finishing write tid 3 to smithi05930959-3 2024-04-19T02:03:40.214 INFO:tasks.rados.rados.0.smithi059.stdout:227: finishing write tid 4 to smithi05930959-3 2024-04-19T02:03:40.214 INFO:tasks.rados.rados.0.smithi059.stdout:227: finishing write tid 5 to smithi05930959-3 2024-04-19T02:03:40.214 INFO:tasks.rados.rados.0.smithi059.stdout:227: finishing write tid 6 to smithi05930959-3 2024-04-19T02:03:40.214 INFO:tasks.rados.rados.0.smithi059.stdout:227: oid 3 updating version 0 to 58 2024-04-19T02:03:40.214 INFO:tasks.rados.rados.0.smithi059.stdout:227: oid 3 version 58 is already newer than 57 2024-04-19T02:03:40.214 INFO:tasks.rados.rados.0.smithi059.stdout:227: oid 3 updating version 58 to 60 2024-04-19T02:03:40.214 INFO:tasks.rados.rados.0.smithi059.stdout:227: oid 3 version 60 is already newer than 59 2024-04-19T02:03:40.214 INFO:tasks.rados.rados.0.smithi059.stdout:227: oid 3 updating version 60 to 61 2024-04-19T02:03:40.214 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 3 v 61 (ObjNum 108 snap 15 seq_num 108) dirty exists 2024-04-19T02:03:40.214 INFO:tasks.rados.rados.0.smithi059.stdout:227: left oid 3 (ObjNum 108 snap 15 seq_num 108) 2024-04-19T02:03:40.226 INFO:tasks.rados.rados.0.smithi059.stdout:227: done (2 left) 2024-04-19T02:03:40.226 INFO:tasks.rados.rados.0.smithi059.stdout:229: done (1 left) 2024-04-19T02:03:40.226 INFO:tasks.rados.rados.0.smithi059.stdout:230: delete oid 11 current snap is 15 2024-04-19T02:03:40.227 INFO:tasks.rados.rados.0.smithi059.stdout:228: finishing write tid 1 to smithi05930959-46 2024-04-19T02:03:40.228 INFO:tasks.rados.rados.0.smithi059.stdout:228: finishing write tid 2 to smithi05930959-46 2024-04-19T02:03:40.229 INFO:tasks.rados.rados.0.smithi059.stdout:228: finishing write tid 3 to smithi05930959-46 2024-04-19T02:03:40.231 INFO:tasks.rados.rados.0.smithi059.stdout:228: finishing write tid 4 to smithi05930959-46 2024-04-19T02:03:40.232 INFO:tasks.rados.rados.0.smithi059.stdout:230: done (1 left) 2024-04-19T02:03:40.233 INFO:tasks.rados.rados.0.smithi059.stdout:231: read oid 45 snap -1 2024-04-19T02:03:40.233 INFO:tasks.rados.rados.0.smithi059.stdout:231: expect (ObjNum 93 snap 9 seq_num 93) 2024-04-19T02:03:40.233 INFO:tasks.rados.rados.0.smithi059.stdout:232: rollback oid 24 current snap is 15 2024-04-19T02:03:40.233 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 24 to 10 2024-04-19T02:03:40.233 INFO:tasks.rados.rados.0.smithi059.stdout:233: rmattr oid 12 current snap is 15 2024-04-19T02:03:40.233 INFO:tasks.rados.rados.0.smithi059.stdout:233: done (3 left) 2024-04-19T02:03:40.233 INFO:tasks.rados.rados.0.smithi059.stdout:234: snap_remove snap 13 2024-04-19T02:03:40.234 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:40.234 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-04-19T02:03:40.234 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-04-19T02:03:40.234 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-04-19T02:03:40.234 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.234 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.4s1 42/26 ECSubWrite(tid=44, reqid=osd.1.0:44, at_version=42'32, trim_to=0'0, roll_forward_to=40'28))", 2024-04-19T02:03:40.235 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.738509+0000", 2024-04-19T02:03:40.235 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 13.495192801, 2024-04-19T02:03:40.235 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.01226059, 2024-04-19T02:03:40.235 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.235 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.235 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.235 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.235 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.235 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.235 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.738509+0000", 2024-04-19T02:03:40.235 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.235 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.235 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.235 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.236 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.738509+0000", 2024-04-19T02:03:40.236 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.236 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.236 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.236 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.236 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.738511+0000", 2024-04-19T02:03:40.236 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.84e-06 2024-04-19T02:03:40.236 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.236 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.236 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.236 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.738565+0000", 2024-04-19T02:03:40.236 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3643000000000001e-05 2024-04-19T02:03:40.236 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.236 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.237 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.237 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.738593+0000", 2024-04-19T02:03:40.237 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8810999999999999e-05 2024-04-19T02:03:40.237 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.237 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.237 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.237 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.738615+0000", 2024-04-19T02:03:40.237 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1421999999999999e-05 2024-04-19T02:03:40.237 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.237 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.237 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.237 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.739435+0000", 2024-04-19T02:03:40.238 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00081972100000000003 2024-04-19T02:03:40.238 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.238 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.238 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.238 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.739460+0000", 2024-04-19T02:03:40.238 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5646000000000001e-05 2024-04-19T02:03:40.238 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.238 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.238 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.238 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.750672+0000", 2024-04-19T02:03:40.238 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011211563000000001 2024-04-19T02:03:40.238 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.239 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.239 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.239 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.750770+0000", 2024-04-19T02:03:40.239 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7943999999999996e-05 2024-04-19T02:03:40.239 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.239 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.239 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.239 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.239 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.239 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.ds4 42/26 ECSubWrite(tid=163, reqid=osd.8.0:163, at_version=42'14, trim_to=0'0, roll_forward_to=39'12))", 2024-04-19T02:03:40.239 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.742684+0000", 2024-04-19T02:03:40.239 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 13.491017854000001, 2024-04-19T02:03:40.239 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012960826, 2024-04-19T02:03:40.239 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.239 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.239 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.240 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.240 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.240 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.240 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.742684+0000", 2024-04-19T02:03:40.240 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.240 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.240 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.240 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.240 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.742684+0000", 2024-04-19T02:03:40.240 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.240 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.240 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.240 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.240 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.742685+0000", 2024-04-19T02:03:40.240 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0999999999999998e-07 2024-04-19T02:03:40.241 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.241 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.241 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.241 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.742739+0000", 2024-04-19T02:03:40.241 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4089999999999999e-05 2024-04-19T02:03:40.241 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.241 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.241 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.241 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.742770+0000", 2024-04-19T02:03:40.241 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0580999999999997e-05 2024-04-19T02:03:40.241 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.241 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.242 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.242 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.742792+0000", 2024-04-19T02:03:40.242 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2985000000000001e-05 2024-04-19T02:03:40.242 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.242 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.242 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.242 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.743203+0000", 2024-04-19T02:03:40.242 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000410379 2024-04-19T02:03:40.242 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.242 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.242 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.242 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.743230+0000", 2024-04-19T02:03:40.242 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6848000000000001e-05 2024-04-19T02:03:40.242 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.243 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.243 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.243 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755545+0000", 2024-04-19T02:03:40.243 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012315609 2024-04-19T02:03:40.243 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.243 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.243 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.243 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755645+0000", 2024-04-19T02:03:40.243 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9623999999999993e-05 2024-04-19T02:03:40.243 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.243 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.243 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.243 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.243 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.244 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.2s1 42/26 ECSubWrite(tid=132, reqid=osd.3.0:132, at_version=42'63, trim_to=0'0, roll_forward_to=40'61))", 2024-04-19T02:03:40.244 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.746263+0000", 2024-04-19T02:03:40.244 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 13.487438989999999, 2024-04-19T02:03:40.244 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013143470000000001, 2024-04-19T02:03:40.244 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.244 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.244 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.244 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.244 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.244 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.244 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746263+0000", 2024-04-19T02:03:40.244 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.245 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.245 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.245 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.245 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746263+0000", 2024-04-19T02:03:40.245 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.245 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.245 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.245 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.245 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746264+0000", 2024-04-19T02:03:40.245 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9399999999999993e-07 2024-04-19T02:03:40.245 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.245 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.245 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.245 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746323+0000", 2024-04-19T02:03:40.245 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8686e-05 2024-04-19T02:03:40.246 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.246 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.246 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.246 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746353+0000", 2024-04-19T02:03:40.246 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0394000000000001e-05 2024-04-19T02:03:40.246 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.246 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.246 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.246 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746375+0000", 2024-04-19T02:03:40.246 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1934999999999999e-05 2024-04-19T02:03:40.246 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.246 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.246 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.246 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746786+0000", 2024-04-19T02:03:40.247 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00041093199999999997 2024-04-19T02:03:40.247 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.247 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.247 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.247 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746812+0000", 2024-04-19T02:03:40.247 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6392000000000001e-05 2024-04-19T02:03:40.247 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.247 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.247 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.247 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.759279+0000", 2024-04-19T02:03:40.247 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012466589 2024-04-19T02:03:40.247 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.247 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.247 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.247 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.759406+0000", 2024-04-19T02:03:40.248 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012754800000000001 2024-04-19T02:03:40.248 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.248 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.248 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.248 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.248 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.248 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.8s4 42/26 ECSubWrite(tid=71, reqid=osd.4.0:71, at_version=42'33, trim_to=0'0, roll_forward_to=37'31))", 2024-04-19T02:03:40.248 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.746410+0000", 2024-04-19T02:03:40.248 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 13.487291986000001, 2024-04-19T02:03:40.248 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012849090000000001, 2024-04-19T02:03:40.248 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.248 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.248 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.248 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.249 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.249 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.249 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746410+0000", 2024-04-19T02:03:40.249 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.249 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.249 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.249 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.249 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746410+0000", 2024-04-19T02:03:40.249 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.249 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.249 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.249 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.249 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746411+0000", 2024-04-19T02:03:40.249 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9000000000000005e-07 2024-04-19T02:03:40.249 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.249 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.250 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.250 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746467+0000", 2024-04-19T02:03:40.250 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6149000000000001e-05 2024-04-19T02:03:40.250 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.250 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.250 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.250 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746511+0000", 2024-04-19T02:03:40.250 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4317000000000003e-05 2024-04-19T02:03:40.250 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.250 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.251 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.251 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746535+0000", 2024-04-19T02:03:40.251 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3655e-05 2024-04-19T02:03:40.251 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.251 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.251 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.251 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.747876+0000", 2024-04-19T02:03:40.251 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0013412490000000001 2024-04-19T02:03:40.251 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.251 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.251 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.252 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.747901+0000", 2024-04-19T02:03:40.252 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4572000000000001e-05 2024-04-19T02:03:40.252 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.252 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.252 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.252 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.759159+0000", 2024-04-19T02:03:40.252 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011258351 2024-04-19T02:03:40.252 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.252 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.252 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.252 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.759259+0000", 2024-04-19T02:03:40.252 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9807000000000002e-05 2024-04-19T02:03:40.252 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.252 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.252 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.253 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.253 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.253 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.as4 42/26 ECSubWrite(tid=82, reqid=osd.11.0:82, at_version=42'34, trim_to=0'0, roll_forward_to=39'32))", 2024-04-19T02:03:40.253 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.749301+0000", 2024-04-19T02:03:40.253 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 13.484400779, 2024-04-19T02:03:40.253 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.015619305999999999, 2024-04-19T02:03:40.253 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.253 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.253 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.253 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.253 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.253 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.254 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749301+0000", 2024-04-19T02:03:40.254 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.254 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.254 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.254 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.254 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749301+0000", 2024-04-19T02:03:40.254 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.254 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.254 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.254 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.254 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749302+0000", 2024-04-19T02:03:40.254 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5000000000000002e-07 2024-04-19T02:03:40.254 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.254 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.255 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.255 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749357+0000", 2024-04-19T02:03:40.255 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5013000000000003e-05 2024-04-19T02:03:40.255 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.255 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.255 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.255 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749391+0000", 2024-04-19T02:03:40.255 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3821e-05 2024-04-19T02:03:40.255 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.255 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.255 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.255 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749414+0000", 2024-04-19T02:03:40.255 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3584999999999999e-05 2024-04-19T02:03:40.255 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.255 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.255 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.256 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.750335+0000", 2024-04-19T02:03:40.256 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00092032799999999999 2024-04-19T02:03:40.256 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.256 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.256 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.256 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.750362+0000", 2024-04-19T02:03:40.256 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7217999999999999e-05 2024-04-19T02:03:40.256 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.256 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.256 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.256 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.764821+0000", 2024-04-19T02:03:40.256 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014459520999999999 2024-04-19T02:03:40.256 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.256 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.256 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.256 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.764921+0000", 2024-04-19T02:03:40.257 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9170000000000001e-05 2024-04-19T02:03:40.257 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.257 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.257 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.257 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.257 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.257 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.5s7 42/26 ECSubWrite(tid=51, reqid=osd.10.0:51, at_version=42'27, trim_to=0'0, roll_forward_to=39'25))", 2024-04-19T02:03:40.257 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.749382+0000", 2024-04-19T02:03:40.257 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 13.484319541, 2024-04-19T02:03:40.257 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010066832, 2024-04-19T02:03:40.257 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.257 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.257 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.257 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.258 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.258 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.258 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749382+0000", 2024-04-19T02:03:40.258 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.258 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.258 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.258 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.258 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749382+0000", 2024-04-19T02:03:40.258 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.258 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.258 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.258 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.258 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749383+0000", 2024-04-19T02:03:40.258 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7000000000000003e-07 2024-04-19T02:03:40.259 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.259 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.259 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.259 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749433+0000", 2024-04-19T02:03:40.259 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9812999999999999e-05 2024-04-19T02:03:40.259 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.259 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.259 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.259 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749459+0000", 2024-04-19T02:03:40.259 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5565e-05 2024-04-19T02:03:40.259 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.259 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.259 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.259 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749478+0000", 2024-04-19T02:03:40.260 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9216e-05 2024-04-19T02:03:40.260 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.260 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.260 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.260 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749657+0000", 2024-04-19T02:03:40.260 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017872799999999999 2024-04-19T02:03:40.260 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.260 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.260 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.260 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749682+0000", 2024-04-19T02:03:40.260 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5734000000000001e-05 2024-04-19T02:03:40.260 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.260 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.260 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.261 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.759360+0000", 2024-04-19T02:03:40.261 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0096778550000000008 2024-04-19T02:03:40.261 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.261 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.261 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.261 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.759449+0000", 2024-04-19T02:03:40.261 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8950999999999996e-05 2024-04-19T02:03:40.261 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.261 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.261 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.261 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.261 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.262 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.2s1 42/26 ECSubWrite(tid=134, reqid=osd.3.0:134, at_version=42'65, trim_to=0'0, roll_forward_to=40'61))", 2024-04-19T02:03:40.262 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.749777+0000", 2024-04-19T02:03:40.262 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 13.483924976999999, 2024-04-19T02:03:40.262 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.015233401000000001, 2024-04-19T02:03:40.262 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.262 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.262 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.262 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.262 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.262 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.262 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749777+0000", 2024-04-19T02:03:40.262 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.263 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.263 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.263 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.263 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749777+0000", 2024-04-19T02:03:40.263 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.263 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.263 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.263 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.263 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749778+0000", 2024-04-19T02:03:40.263 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.336e-06 2024-04-19T02:03:40.263 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.264 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.264 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.264 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749823+0000", 2024-04-19T02:03:40.264 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4277000000000003e-05 2024-04-19T02:03:40.264 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.264 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.264 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.264 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749845+0000", 2024-04-19T02:03:40.264 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2677999999999999e-05 2024-04-19T02:03:40.264 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.264 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.264 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.264 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749864+0000", 2024-04-19T02:03:40.264 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8913e-05 2024-04-19T02:03:40.264 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.265 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.265 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.265 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.753987+0000", 2024-04-19T02:03:40.265 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0041232070000000003 2024-04-19T02:03:40.265 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.265 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.265 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.265 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.754010+0000", 2024-04-19T02:03:40.265 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2792000000000001e-05 2024-04-19T02:03:40.265 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.265 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.265 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.265 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.764933+0000", 2024-04-19T02:03:40.265 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.01092277 2024-04-19T02:03:40.265 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.266 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.266 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.266 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.765010+0000", 2024-04-19T02:03:40.266 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7428000000000005e-05 2024-04-19T02:03:40.266 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.266 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.266 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.266 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.266 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.266 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.es2 42/26 ECSubWrite(tid=32, reqid=osd.7.0:32, at_version=42'25, trim_to=0'0, roll_forward_to=40'23))", 2024-04-19T02:03:40.267 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.752167+0000", 2024-04-19T02:03:40.267 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 13.481534993, 2024-04-19T02:03:40.267 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012865329, 2024-04-19T02:03:40.267 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.267 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.267 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.267 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.267 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.267 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.267 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.752167+0000", 2024-04-19T02:03:40.267 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.267 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.268 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.268 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.268 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.752167+0000", 2024-04-19T02:03:40.268 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.268 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.268 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.268 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.268 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.752168+0000", 2024-04-19T02:03:40.268 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6700000000000003e-07 2024-04-19T02:03:40.268 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.268 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.269 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.269 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.752240+0000", 2024-04-19T02:03:40.269 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1988000000000003e-05 2024-04-19T02:03:40.269 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.269 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.269 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.269 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.752265+0000", 2024-04-19T02:03:40.269 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5626000000000002e-05 2024-04-19T02:03:40.269 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.269 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.269 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.269 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.752287+0000", 2024-04-19T02:03:40.269 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2045e-05 2024-04-19T02:03:40.269 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.270 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.270 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.270 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.752546+0000", 2024-04-19T02:03:40.270 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000258342 2024-04-19T02:03:40.270 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.270 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.270 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.270 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.752571+0000", 2024-04-19T02:03:40.270 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4941999999999999e-05 2024-04-19T02:03:40.270 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.270 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.270 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.270 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.764965+0000", 2024-04-19T02:03:40.270 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012394196999999999 2024-04-19T02:03:40.271 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.271 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.271 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.271 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.765032+0000", 2024-04-19T02:03:40.271 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7621999999999997e-05 2024-04-19T02:03:40.271 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.271 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.271 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.271 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.272 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.272 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.4s1 42/26 ECSubWrite(tid=45, reqid=osd.1.0:44, at_version=0'0, trim_to=0'0, roll_forward_to=42'32))", 2024-04-19T02:03:40.272 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.754514+0000", 2024-04-19T02:03:40.272 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 13.479188298, 2024-04-19T02:03:40.272 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011039706, 2024-04-19T02:03:40.272 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.272 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.272 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.272 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.272 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.272 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.272 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.754514+0000", 2024-04-19T02:03:40.272 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.272 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.273 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.273 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.273 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.754514+0000", 2024-04-19T02:03:40.273 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.273 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.273 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.273 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.273 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.754516+0000", 2024-04-19T02:03:40.273 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8560000000000001e-06 2024-04-19T02:03:40.273 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.273 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.273 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.273 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.754544+0000", 2024-04-19T02:03:40.273 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8248000000000001e-05 2024-04-19T02:03:40.273 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.273 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.274 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.274 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.754574+0000", 2024-04-19T02:03:40.274 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9935000000000001e-05 2024-04-19T02:03:40.274 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.274 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.274 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.274 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.754595+0000", 2024-04-19T02:03:40.274 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1596000000000001e-05 2024-04-19T02:03:40.274 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.274 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.274 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.274 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.754758+0000", 2024-04-19T02:03:40.274 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016278500000000001 2024-04-19T02:03:40.274 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.274 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.275 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.275 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.754784+0000", 2024-04-19T02:03:40.275 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5698e-05 2024-04-19T02:03:40.275 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.275 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.275 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.275 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.765464+0000", 2024-04-19T02:03:40.275 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010680044 2024-04-19T02:03:40.275 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.275 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.275 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.275 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.765553+0000", 2024-04-19T02:03:40.275 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.9543999999999995e-05 2024-04-19T02:03:40.275 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.275 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.275 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.276 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.276 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.276 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.fs4 42/26 ECSubWrite(tid=83, reqid=osd.11.0:83, at_version=42'26, trim_to=0'0, roll_forward_to=39'24))", 2024-04-19T02:03:40.276 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.755206+0000", 2024-04-19T02:03:40.276 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 13.478495785, 2024-04-19T02:03:40.276 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0097280309999999998, 2024-04-19T02:03:40.276 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.276 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.276 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.276 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.276 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.276 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.276 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755206+0000", 2024-04-19T02:03:40.276 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.276 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.276 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.277 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.277 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755206+0000", 2024-04-19T02:03:40.277 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.277 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.277 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.277 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.277 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755208+0000", 2024-04-19T02:03:40.277 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9400000000000001e-06 2024-04-19T02:03:40.277 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.277 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.277 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.277 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755258+0000", 2024-04-19T02:03:40.277 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0040000000000002e-05 2024-04-19T02:03:40.277 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.277 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.277 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.278 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755289+0000", 2024-04-19T02:03:40.278 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0368000000000001e-05 2024-04-19T02:03:40.278 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.278 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.278 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.278 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755311+0000", 2024-04-19T02:03:40.278 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2430999999999999e-05 2024-04-19T02:03:40.278 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.278 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.278 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.278 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755525+0000", 2024-04-19T02:03:40.278 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000214513 2024-04-19T02:03:40.278 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.278 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.279 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.279 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755550+0000", 2024-04-19T02:03:40.279 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4754999999999999e-05 2024-04-19T02:03:40.279 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.279 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.279 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.279 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.764835+0000", 2024-04-19T02:03:40.279 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.009284647 2024-04-19T02:03:40.279 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.279 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.279 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.279 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.764934+0000", 2024-04-19T02:03:40.279 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9337000000000001e-05 2024-04-19T02:03:40.279 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.279 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.279 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.279 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.280 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.280 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.ds4 46/26 ECSubWrite(tid=187, reqid=osd.8.0:187, at_version=46'35, trim_to=0'0, roll_forward_to=45'31))", 2024-04-19T02:03:40.280 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.058841+0000", 2024-04-19T02:03:40.280 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 10.174860835, 2024-04-19T02:03:40.280 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0093001779999999992, 2024-04-19T02:03:40.280 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.280 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.280 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.280 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.280 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.280 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.280 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.058841+0000", 2024-04-19T02:03:40.280 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.280 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.280 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.281 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.281 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.058841+0000", 2024-04-19T02:03:40.281 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.281 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.281 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.281 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.281 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.058843+0000", 2024-04-19T02:03:40.281 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.81e-06 2024-04-19T02:03:40.281 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.281 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.281 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.281 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.058892+0000", 2024-04-19T02:03:40.281 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9333999999999999e-05 2024-04-19T02:03:40.281 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.281 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.281 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.282 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.058927+0000", 2024-04-19T02:03:40.282 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4357999999999999e-05 2024-04-19T02:03:40.282 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.282 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.282 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.282 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.058950+0000", 2024-04-19T02:03:40.282 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3261e-05 2024-04-19T02:03:40.282 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.282 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.282 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.282 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.060130+0000", 2024-04-19T02:03:40.282 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0011804249999999999 2024-04-19T02:03:40.282 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.282 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.282 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.283 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.060154+0000", 2024-04-19T02:03:40.283 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4111999999999999e-05 2024-04-19T02:03:40.283 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.283 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.283 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.283 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.068068+0000", 2024-04-19T02:03:40.283 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0079134419999999997 2024-04-19T02:03:40.283 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.283 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.283 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.283 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.068141+0000", 2024-04-19T02:03:40.283 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3436000000000006e-05 2024-04-19T02:03:40.283 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.283 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.283 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.283 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.284 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.284 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.as4 46/26 ECSubWrite(tid=103, reqid=osd.11.0:103, at_version=46'49, trim_to=0'0, roll_forward_to=45'45))", 2024-04-19T02:03:40.284 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.067936+0000", 2024-04-19T02:03:40.284 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 10.165765591, 2024-04-19T02:03:40.284 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.017806114000000001, 2024-04-19T02:03:40.284 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.284 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.284 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.284 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.284 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.284 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.284 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.067936+0000", 2024-04-19T02:03:40.284 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.284 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.284 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.285 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.285 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.067936+0000", 2024-04-19T02:03:40.285 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.285 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.285 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.285 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.285 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.067938+0000", 2024-04-19T02:03:40.285 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.313e-06 2024-04-19T02:03:40.285 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.285 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.285 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.285 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.067990+0000", 2024-04-19T02:03:40.285 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2490000000000001e-05 2024-04-19T02:03:40.285 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.285 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.286 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.286 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.068014+0000", 2024-04-19T02:03:40.286 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4172000000000002e-05 2024-04-19T02:03:40.286 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.286 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.286 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.286 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.068034+0000", 2024-04-19T02:03:40.286 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9471999999999999e-05 2024-04-19T02:03:40.286 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.286 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.286 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.286 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.069268+0000", 2024-04-19T02:03:40.286 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001234165 2024-04-19T02:03:40.286 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.286 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.286 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.287 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.069293+0000", 2024-04-19T02:03:40.287 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4702000000000002e-05 2024-04-19T02:03:40.287 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.287 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.287 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.287 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.085653+0000", 2024-04-19T02:03:40.287 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.016360535999999998 2024-04-19T02:03:40.287 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.287 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.287 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.287 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.085743+0000", 2024-04-19T02:03:40.287 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.9264000000000005e-05 2024-04-19T02:03:40.287 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.287 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.287 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.288 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.288 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.288 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.1s3 46/26 ECSubWrite(tid=20, reqid=client.4509.0:415, at_version=46'8, trim_to=0'0, roll_forward_to=29'6))", 2024-04-19T02:03:40.288 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.077639+0000", 2024-04-19T02:03:40.288 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 10.156062921, 2024-04-19T02:03:40.288 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010858876999999999, 2024-04-19T02:03:40.288 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.288 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.288 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.288 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.288 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.288 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.288 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.077639+0000", 2024-04-19T02:03:40.288 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.288 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.289 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.289 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.289 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.077639+0000", 2024-04-19T02:03:40.289 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.289 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.289 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.289 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.289 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.077640+0000", 2024-04-19T02:03:40.289 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1999999999999999e-07 2024-04-19T02:03:40.289 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.289 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.289 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.289 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078039+0000", 2024-04-19T02:03:40.289 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00039909899999999998 2024-04-19T02:03:40.289 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.290 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.290 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.290 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078071+0000", 2024-04-19T02:03:40.290 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1987999999999999e-05 2024-04-19T02:03:40.290 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.290 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.290 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.290 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078093+0000", 2024-04-19T02:03:40.290 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2086000000000002e-05 2024-04-19T02:03:40.290 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.290 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.290 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.290 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078730+0000", 2024-04-19T02:03:40.290 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00063664699999999995 2024-04-19T02:03:40.290 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.291 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.291 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.291 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078755+0000", 2024-04-19T02:03:40.291 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5029e-05 2024-04-19T02:03:40.291 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.291 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.291 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.291 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.088431+0000", 2024-04-19T02:03:40.291 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0096761820000000002 2024-04-19T02:03:40.291 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.291 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.291 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.291 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.088498+0000", 2024-04-19T02:03:40.291 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7125999999999994e-05 2024-04-19T02:03:40.291 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.291 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.292 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.292 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.292 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.292 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.2s1 46/26 ECSubWrite(tid=159, reqid=osd.3.0:159, at_version=46'75, trim_to=0'0, roll_forward_to=45'73))", 2024-04-19T02:03:40.292 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.078187+0000", 2024-04-19T02:03:40.292 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 10.155514953999999, 2024-04-19T02:03:40.292 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0097959210000000008, 2024-04-19T02:03:40.292 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.292 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.292 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.292 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.292 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.292 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.292 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078187+0000", 2024-04-19T02:03:40.292 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.292 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.293 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.293 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.293 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078187+0000", 2024-04-19T02:03:40.293 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.293 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.293 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.293 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.293 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078188+0000", 2024-04-19T02:03:40.293 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.39e-06 2024-04-19T02:03:40.293 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.293 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.293 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.293 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078235+0000", 2024-04-19T02:03:40.293 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6297000000000001e-05 2024-04-19T02:03:40.293 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.294 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.294 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.294 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078262+0000", 2024-04-19T02:03:40.294 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7399e-05 2024-04-19T02:03:40.294 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.294 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.294 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.294 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078282+0000", 2024-04-19T02:03:40.294 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9908e-05 2024-04-19T02:03:40.294 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.294 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.294 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.294 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.079243+0000", 2024-04-19T02:03:40.294 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00096095200000000005 2024-04-19T02:03:40.294 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.295 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.295 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.295 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.079266+0000", 2024-04-19T02:03:40.295 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2994999999999999e-05 2024-04-19T02:03:40.295 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.295 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.295 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.295 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.087819+0000", 2024-04-19T02:03:40.295 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0085527599999999995 2024-04-19T02:03:40.295 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.295 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.295 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.295 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.087983+0000", 2024-04-19T02:03:40.295 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016422 2024-04-19T02:03:40.295 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.296 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.296 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.296 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.296 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.296 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.2s1 46/26 ECSubWrite(tid=163, reqid=client.4509.0:425, at_version=46'78, trim_to=0'0, roll_forward_to=46'75))", 2024-04-19T02:03:40.296 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.196489+0000", 2024-04-19T02:03:40.296 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 10.037212972000001, 2024-04-19T02:03:40.296 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013644882000000001, 2024-04-19T02:03:40.296 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.296 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.296 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.296 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.296 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.296 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.296 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196489+0000", 2024-04-19T02:03:40.297 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.297 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.297 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.297 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.297 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196489+0000", 2024-04-19T02:03:40.297 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.297 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.297 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.297 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.297 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196491+0000", 2024-04-19T02:03:40.297 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4269999999999998e-06 2024-04-19T02:03:40.297 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.297 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.297 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.297 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196543+0000", 2024-04-19T02:03:40.298 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1112999999999997e-05 2024-04-19T02:03:40.298 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.298 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.298 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.298 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196607+0000", 2024-04-19T02:03:40.298 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4183000000000005e-05 2024-04-19T02:03:40.298 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.298 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.298 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.298 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196634+0000", 2024-04-19T02:03:40.298 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6968e-05 2024-04-19T02:03:40.298 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.298 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.298 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.298 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.198568+0000", 2024-04-19T02:03:40.299 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0019338129999999999 2024-04-19T02:03:40.299 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.299 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.299 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.299 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.198591+0000", 2024-04-19T02:03:40.299 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3898000000000001e-05 2024-04-19T02:03:40.299 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.299 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.299 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.299 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.210049+0000", 2024-04-19T02:03:40.299 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011457355000000001 2024-04-19T02:03:40.299 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.299 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.299 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.299 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.210134+0000", 2024-04-19T02:03:40.300 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5124999999999996e-05 2024-04-19T02:03:40.300 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.300 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.300 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.300 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.300 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.300 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.ds4 46/26 ECSubWrite(tid=190, reqid=client.4509.0:428, at_version=46'37, trim_to=0'0, roll_forward_to=46'35))", 2024-04-19T02:03:40.300 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.196834+0000", 2024-04-19T02:03:40.300 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 10.036867688999999, 2024-04-19T02:03:40.300 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011503825000000001, 2024-04-19T02:03:40.300 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.300 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.300 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.300 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.300 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.300 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.301 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196834+0000", 2024-04-19T02:03:40.301 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.301 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.301 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.301 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.301 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196834+0000", 2024-04-19T02:03:40.301 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.301 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.301 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.301 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.301 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196835+0000", 2024-04-19T02:03:40.301 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.13e-06 2024-04-19T02:03:40.301 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.301 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.301 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.302 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196895+0000", 2024-04-19T02:03:40.302 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9481999999999997e-05 2024-04-19T02:03:40.302 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.302 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.302 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.302 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196922+0000", 2024-04-19T02:03:40.302 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6639e-05 2024-04-19T02:03:40.302 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.302 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.302 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.302 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196942+0000", 2024-04-19T02:03:40.302 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0931999999999999e-05 2024-04-19T02:03:40.302 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.302 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.302 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.303 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.197738+0000", 2024-04-19T02:03:40.303 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00079601499999999996 2024-04-19T02:03:40.303 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.303 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.303 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.303 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.197811+0000", 2024-04-19T02:03:40.303 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2916000000000005e-05 2024-04-19T02:03:40.303 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.303 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.303 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.303 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.208200+0000", 2024-04-19T02:03:40.303 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010388246 2024-04-19T02:03:40.303 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.303 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.303 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.304 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.208338+0000", 2024-04-19T02:03:40.304 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000138465 2024-04-19T02:03:40.304 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.304 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.304 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.304 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.304 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.304 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.2s1 46/26 ECSubWrite(tid=164, reqid=client.4509.0:426, at_version=46'79, trim_to=0'0, roll_forward_to=46'75))", 2024-04-19T02:03:40.304 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.199712+0000", 2024-04-19T02:03:40.304 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 10.033990284, 2024-04-19T02:03:40.304 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010489036, 2024-04-19T02:03:40.304 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.304 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.304 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.304 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.305 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.305 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.305 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.199712+0000", 2024-04-19T02:03:40.305 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.305 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.305 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.305 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.305 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.199712+0000", 2024-04-19T02:03:40.305 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.305 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.305 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.305 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.305 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.199713+0000", 2024-04-19T02:03:40.305 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7630000000000001e-06 2024-04-19T02:03:40.305 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.305 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.306 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.306 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.199758+0000", 2024-04-19T02:03:40.306 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4969999999999998e-05 2024-04-19T02:03:40.306 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.306 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.306 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.306 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.199784+0000", 2024-04-19T02:03:40.306 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5846e-05 2024-04-19T02:03:40.306 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.306 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.306 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.306 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.199805+0000", 2024-04-19T02:03:40.306 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0271999999999998e-05 2024-04-19T02:03:40.306 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.306 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.307 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.307 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.199970+0000", 2024-04-19T02:03:40.307 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016560899999999999 2024-04-19T02:03:40.307 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.307 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.307 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.307 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.199993+0000", 2024-04-19T02:03:40.307 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2549e-05 2024-04-19T02:03:40.307 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.307 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.307 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.307 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.210142+0000", 2024-04-19T02:03:40.307 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010149768 2024-04-19T02:03:40.307 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.307 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.307 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.308 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.210201+0000", 2024-04-19T02:03:40.308 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8258999999999999e-05 2024-04-19T02:03:40.308 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.308 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.308 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.308 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.308 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.308 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.2s1 47/26 ECSubWrite(tid=182, reqid=osd.3.0:182, at_version=47'89, trim_to=0'0, roll_forward_to=46'85))", 2024-04-19T02:03:40.308 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.081506+0000", 2024-04-19T02:03:40.308 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 9.1521963700000004, 2024-04-19T02:03:40.308 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0089814260000000007, 2024-04-19T02:03:40.308 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.308 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.308 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.308 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.308 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.309 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.309 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081506+0000", 2024-04-19T02:03:40.309 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.309 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.309 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.309 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.309 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081506+0000", 2024-04-19T02:03:40.309 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.309 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.309 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.309 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.309 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081507+0000", 2024-04-19T02:03:40.309 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7889999999999999e-06 2024-04-19T02:03:40.309 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.310 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.310 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.310 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081561+0000", 2024-04-19T02:03:40.310 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3247e-05 2024-04-19T02:03:40.310 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.310 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.310 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.310 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081593+0000", 2024-04-19T02:03:40.310 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.2187999999999997e-05 2024-04-19T02:03:40.310 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.310 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.310 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.310 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081615+0000", 2024-04-19T02:03:40.310 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1892e-05 2024-04-19T02:03:40.310 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.311 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.311 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.311 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081817+0000", 2024-04-19T02:03:40.311 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000202351 2024-04-19T02:03:40.311 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.311 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.311 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.311 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081841+0000", 2024-04-19T02:03:40.311 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4189000000000002e-05 2024-04-19T02:03:40.311 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.311 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.311 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.311 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.090389+0000", 2024-04-19T02:03:40.311 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0085477729999999998 2024-04-19T02:03:40.311 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.311 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.312 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.312 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.090487+0000", 2024-04-19T02:03:40.312 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7997000000000004e-05 2024-04-19T02:03:40.312 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.312 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.312 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.312 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.312 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.312 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.1s3 47/26 ECSubWrite(tid=30, reqid=osd.0.0:30, at_version=47'17, trim_to=0'0, roll_forward_to=46'13))", 2024-04-19T02:03:40.312 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.085565+0000", 2024-04-19T02:03:40.312 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 9.1481368700000001, 2024-04-19T02:03:40.312 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010262217000000001, 2024-04-19T02:03:40.312 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.312 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.312 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.313 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.313 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.313 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.313 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085565+0000", 2024-04-19T02:03:40.313 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.313 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.313 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.313 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.313 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085565+0000", 2024-04-19T02:03:40.313 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.313 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.313 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.313 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.313 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085566+0000", 2024-04-19T02:03:40.313 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.33e-07 2024-04-19T02:03:40.313 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.314 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.314 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.314 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085617+0000", 2024-04-19T02:03:40.314 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0487e-05 2024-04-19T02:03:40.314 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.314 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.314 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.314 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085639+0000", 2024-04-19T02:03:40.314 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2866000000000001e-05 2024-04-19T02:03:40.314 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.314 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.314 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.314 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085658+0000", 2024-04-19T02:03:40.314 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8575000000000001e-05 2024-04-19T02:03:40.314 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.315 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.315 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.315 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.090331+0000", 2024-04-19T02:03:40.315 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0046729650000000003 2024-04-19T02:03:40.315 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.315 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.315 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.315 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.090358+0000", 2024-04-19T02:03:40.315 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7424999999999999e-05 2024-04-19T02:03:40.315 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.315 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.315 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.315 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.095729+0000", 2024-04-19T02:03:40.315 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0053710989999999998 2024-04-19T02:03:40.315 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.316 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.316 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.316 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.095827+0000", 2024-04-19T02:03:40.316 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7867000000000003e-05 2024-04-19T02:03:40.316 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.316 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.316 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.316 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.316 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.316 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.8s4 47/26 ECSubWrite(tid=86, reqid=osd.4.0:86, at_version=47'44, trim_to=0'0, roll_forward_to=46'42))", 2024-04-19T02:03:40.316 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.090878+0000", 2024-04-19T02:03:40.316 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 9.1428238969999995, 2024-04-19T02:03:40.316 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0091567460000000003, 2024-04-19T02:03:40.316 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.316 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.317 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.317 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.317 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.317 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.317 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.090878+0000", 2024-04-19T02:03:40.317 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.317 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.317 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.317 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.317 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.090878+0000", 2024-04-19T02:03:40.317 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.317 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.317 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.317 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.317 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.090879+0000", 2024-04-19T02:03:40.317 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4000000000000001e-07 2024-04-19T02:03:40.318 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.318 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.318 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.318 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.090930+0000", 2024-04-19T02:03:40.318 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1196999999999998e-05 2024-04-19T02:03:40.318 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.318 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.318 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.318 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.090958+0000", 2024-04-19T02:03:40.318 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8041000000000001e-05 2024-04-19T02:03:40.318 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.318 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.318 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.318 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.090979+0000", 2024-04-19T02:03:40.318 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1302999999999999e-05 2024-04-19T02:03:40.319 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.319 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.319 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.319 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.091142+0000", 2024-04-19T02:03:40.319 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016285 2024-04-19T02:03:40.319 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.319 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.319 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.319 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.091167+0000", 2024-04-19T02:03:40.319 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5242e-05 2024-04-19T02:03:40.319 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.319 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.319 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.319 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.099946+0000", 2024-04-19T02:03:40.319 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0087790379999999994 2024-04-19T02:03:40.319 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.320 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.320 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.320 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.100035+0000", 2024-04-19T02:03:40.320 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8435000000000003e-05 2024-04-19T02:03:40.320 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.320 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.320 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.320 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.320 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.320 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:40.322 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-19T02:03:40.322 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:03:40.318+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:03:40.423 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:40.426+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:40.488 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:40.488 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:40.488 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 0 2024-04-19T02:03:40.489 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:40.502 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-19T02:03:40.547 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:40.546+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:40.618 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:40.619 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:40.619 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-04-19T02:03:40.619 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-04-19T02:03:40.619 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:40.624 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:40.622+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:40.633 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-19T02:03:40.725 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:40.726+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:40.794 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:40.794 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-04-19T02:03:40.794 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-04-19T02:03:40.794 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-04-19T02:03:40.794 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.795 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.8s6 37/26 ECSubWrite(tid=65, reqid=client.4509.0:295, at_version=37'29, trim_to=0'0, roll_forward_to=37'28))", 2024-04-19T02:03:40.795 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:21.724488+0000", 2024-04-19T02:03:40.795 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 19.069770028000001, 2024-04-19T02:03:40.795 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0045200860000000004, 2024-04-19T02:03:40.795 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.795 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.795 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.795 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.795 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.795 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.795 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:21.724488+0000", 2024-04-19T02:03:40.795 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.795 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.795 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.796 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.796 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:21.724488+0000", 2024-04-19T02:03:40.796 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.796 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.796 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.796 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.796 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:21.724489+0000", 2024-04-19T02:03:40.796 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1500000000000004e-07 2024-04-19T02:03:40.796 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.796 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.796 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.796 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:21.724614+0000", 2024-04-19T02:03:40.796 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012561500000000001 2024-04-19T02:03:40.796 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.797 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.797 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.797 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:21.724634+0000", 2024-04-19T02:03:40.797 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9160999999999998e-05 2024-04-19T02:03:40.797 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.797 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.797 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.797 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:21.724643+0000", 2024-04-19T02:03:40.797 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7869999999999996e-06 2024-04-19T02:03:40.797 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.797 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.797 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.797 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:21.724837+0000", 2024-04-19T02:03:40.797 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00019329300000000001 2024-04-19T02:03:40.797 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.798 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.798 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.798 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:21.724862+0000", 2024-04-19T02:03:40.798 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5412e-05 2024-04-19T02:03:40.798 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.798 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.798 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.798 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:21.728921+0000", 2024-04-19T02:03:40.798 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0040589140000000003 2024-04-19T02:03:40.798 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.798 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.798 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.798 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:21.729008+0000", 2024-04-19T02:03:40.799 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7188999999999995e-05 2024-04-19T02:03:40.799 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.799 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.799 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.799 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.799 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.799 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.8s6 42/26 ECSubWrite(tid=71, reqid=osd.4.0:71, at_version=42'33, trim_to=0'0, roll_forward_to=37'31))", 2024-04-19T02:03:40.799 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.746437+0000", 2024-04-19T02:03:40.799 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 14.047820973, 2024-04-19T02:03:40.799 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011548248000000001, 2024-04-19T02:03:40.799 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.799 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.799 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.799 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.799 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.799 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.800 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746437+0000", 2024-04-19T02:03:40.800 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.800 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.800 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.800 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.800 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746437+0000", 2024-04-19T02:03:40.800 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.800 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.800 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.800 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.800 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746438+0000", 2024-04-19T02:03:40.800 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6000000000000003e-07 2024-04-19T02:03:40.800 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.800 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.801 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.801 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746491+0000", 2024-04-19T02:03:40.801 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2939000000000003e-05 2024-04-19T02:03:40.801 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.801 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.801 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.801 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746520+0000", 2024-04-19T02:03:40.801 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8835000000000001e-05 2024-04-19T02:03:40.801 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.801 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.801 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.801 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746541+0000", 2024-04-19T02:03:40.801 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1628999999999999e-05 2024-04-19T02:03:40.801 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.801 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.802 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.802 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746712+0000", 2024-04-19T02:03:40.802 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017087100000000001 2024-04-19T02:03:40.802 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.802 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.802 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.802 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746742+0000", 2024-04-19T02:03:40.802 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9691000000000001e-05 2024-04-19T02:03:40.802 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.802 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.802 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.802 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.757878+0000", 2024-04-19T02:03:40.802 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011136546000000001 2024-04-19T02:03:40.803 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.803 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.803 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.803 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.757985+0000", 2024-04-19T02:03:40.803 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000106977 2024-04-19T02:03:40.803 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.803 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.803 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.803 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.803 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.803 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.bs3 42/26 ECSubWrite(tid=133, reqid=osd.3.0:133, at_version=42'21, trim_to=0'0, roll_forward_to=39'19))", 2024-04-19T02:03:40.803 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.748283+0000", 2024-04-19T02:03:40.803 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 14.045975515, 2024-04-19T02:03:40.803 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012079138999999999, 2024-04-19T02:03:40.804 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.804 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.804 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.804 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.804 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.804 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.804 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748283+0000", 2024-04-19T02:03:40.804 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.804 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.804 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.804 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.804 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748283+0000", 2024-04-19T02:03:40.804 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.804 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.804 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.804 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.805 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748283+0000", 2024-04-19T02:03:40.805 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3300000000000002e-07 2024-04-19T02:03:40.805 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.805 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.805 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.805 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748372+0000", 2024-04-19T02:03:40.805 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8783999999999996e-05 2024-04-19T02:03:40.805 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.805 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.805 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.805 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748403+0000", 2024-04-19T02:03:40.805 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1228e-05 2024-04-19T02:03:40.805 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.805 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.805 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.806 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748427+0000", 2024-04-19T02:03:40.806 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3512000000000001e-05 2024-04-19T02:03:40.806 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.806 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.806 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.806 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748811+0000", 2024-04-19T02:03:40.806 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00038409700000000002 2024-04-19T02:03:40.806 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.806 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.806 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.806 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748836+0000", 2024-04-19T02:03:40.806 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5364999999999999e-05 2024-04-19T02:03:40.806 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.806 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.807 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.807 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.760267+0000", 2024-04-19T02:03:40.807 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011430656000000001 2024-04-19T02:03:40.807 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.807 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.807 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.807 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.760362+0000", 2024-04-19T02:03:40.807 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4864000000000005e-05 2024-04-19T02:03:40.807 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.807 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.807 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.807 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.807 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.807 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.bs3 42/26 ECSubWrite(tid=135, reqid=osd.3.0:135, at_version=42'23, trim_to=0'0, roll_forward_to=39'19))", 2024-04-19T02:03:40.807 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.748483+0000", 2024-04-19T02:03:40.808 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 14.045774685, 2024-04-19T02:03:40.808 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.015459633, 2024-04-19T02:03:40.808 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.808 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.808 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.808 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.808 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.808 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.808 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748483+0000", 2024-04-19T02:03:40.808 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.808 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.808 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.808 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.808 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748483+0000", 2024-04-19T02:03:40.809 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.809 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.809 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.809 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.809 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748484+0000", 2024-04-19T02:03:40.809 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8899999999999998e-07 2024-04-19T02:03:40.809 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.809 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.809 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.809 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748527+0000", 2024-04-19T02:03:40.809 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2651000000000002e-05 2024-04-19T02:03:40.809 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.809 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.809 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.809 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748550+0000", 2024-04-19T02:03:40.810 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3345000000000001e-05 2024-04-19T02:03:40.810 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.810 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.810 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.810 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748569+0000", 2024-04-19T02:03:40.810 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8722000000000001e-05 2024-04-19T02:03:40.810 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.810 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.810 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.810 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.752605+0000", 2024-04-19T02:03:40.810 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0040362510000000002 2024-04-19T02:03:40.810 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.810 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.810 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.811 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.752629+0000", 2024-04-19T02:03:40.811 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3685000000000001e-05 2024-04-19T02:03:40.811 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.811 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.811 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.811 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.763825+0000", 2024-04-19T02:03:40.811 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011195577999999999 2024-04-19T02:03:40.811 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.811 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.811 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.811 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.763943+0000", 2024-04-19T02:03:40.812 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000118412 2024-04-19T02:03:40.812 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.812 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.812 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.812 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.812 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.812 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.5s6 42/26 ECSubWrite(tid=51, reqid=osd.10.0:51, at_version=42'27, trim_to=0'0, roll_forward_to=39'25))", 2024-04-19T02:03:40.812 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.749097+0000", 2024-04-19T02:03:40.812 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 14.045160709999999, 2024-04-19T02:03:40.812 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011834341, 2024-04-19T02:03:40.812 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.812 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.813 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.813 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.813 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.813 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.813 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749097+0000", 2024-04-19T02:03:40.813 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.813 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.813 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.813 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.813 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749097+0000", 2024-04-19T02:03:40.813 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.813 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.813 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.813 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.813 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749098+0000", 2024-04-19T02:03:40.814 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9299999999999997e-07 2024-04-19T02:03:40.814 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.814 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.814 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.814 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749170+0000", 2024-04-19T02:03:40.814 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2208999999999999e-05 2024-04-19T02:03:40.814 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.814 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.814 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.814 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749270+0000", 2024-04-19T02:03:40.814 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000100089 2024-04-19T02:03:40.814 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.814 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.814 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.815 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749297+0000", 2024-04-19T02:03:40.815 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6485000000000001e-05 2024-04-19T02:03:40.815 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.815 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.815 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.815 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749537+0000", 2024-04-19T02:03:40.815 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00024036700000000001 2024-04-19T02:03:40.815 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.815 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.815 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.815 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749562+0000", 2024-04-19T02:03:40.816 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5082000000000001e-05 2024-04-19T02:03:40.816 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.816 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.816 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.816 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.760847+0000", 2024-04-19T02:03:40.816 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011285165999999999 2024-04-19T02:03:40.816 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.816 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.816 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.817 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.760932+0000", 2024-04-19T02:03:40.817 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4250000000000001e-05 2024-04-19T02:03:40.817 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.817 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.817 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.817 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.817 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.817 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.5s6 42/26 ECSubWrite(tid=52, reqid=osd.10.0:52, at_version=42'29, trim_to=0'0, roll_forward_to=39'25))", 2024-04-19T02:03:40.817 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.758688+0000", 2024-04-19T02:03:40.817 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 14.035569580000001, 2024-04-19T02:03:40.817 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0063222319999999997, 2024-04-19T02:03:40.817 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.817 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.817 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.818 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.818 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.818 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.818 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.758688+0000", 2024-04-19T02:03:40.818 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.818 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.818 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.818 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.818 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.758688+0000", 2024-04-19T02:03:40.818 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.818 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.818 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.818 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.819 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.758690+0000", 2024-04-19T02:03:40.819 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.81e-06 2024-04-19T02:03:40.819 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.819 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.819 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.819 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.758767+0000", 2024-04-19T02:03:40.819 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7218000000000005e-05 2024-04-19T02:03:40.819 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.819 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.819 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.819 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.758796+0000", 2024-04-19T02:03:40.819 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8538e-05 2024-04-19T02:03:40.819 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.819 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.820 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.820 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.758817+0000", 2024-04-19T02:03:40.820 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1404999999999999e-05 2024-04-19T02:03:40.820 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.820 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.820 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.820 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.758982+0000", 2024-04-19T02:03:40.820 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016446899999999999 2024-04-19T02:03:40.820 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.820 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.820 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.820 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.759014+0000", 2024-04-19T02:03:40.820 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.2407999999999999e-05 2024-04-19T02:03:40.820 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.820 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.821 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.821 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.764947+0000", 2024-04-19T02:03:40.821 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.005932522 2024-04-19T02:03:40.821 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.821 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.821 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.821 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.765011+0000", 2024-04-19T02:03:40.821 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3862000000000006e-05 2024-04-19T02:03:40.821 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.821 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.821 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.821 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.821 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.821 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.8s6 42/26 ECSubWrite(tid=72, reqid=osd.4.0:71, at_version=0'0, trim_to=0'0, roll_forward_to=42'33))", 2024-04-19T02:03:40.821 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.776899+0000", 2024-04-19T02:03:40.822 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 14.017359517999999, 2024-04-19T02:03:40.822 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003554041, 2024-04-19T02:03:40.822 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.822 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.822 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.822 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.822 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.822 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.822 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.776899+0000", 2024-04-19T02:03:40.822 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.823 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.823 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.823 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.823 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.776899+0000", 2024-04-19T02:03:40.823 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.823 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.824 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.824 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.824 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.776900+0000", 2024-04-19T02:03:40.824 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.688e-06 2024-04-19T02:03:40.824 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.824 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.824 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.825 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.776930+0000", 2024-04-19T02:03:40.825 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9767000000000001e-05 2024-04-19T02:03:40.825 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.825 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.825 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.825 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.776958+0000", 2024-04-19T02:03:40.826 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8067000000000001e-05 2024-04-19T02:03:40.826 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.826 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.826 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.826 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.776986+0000", 2024-04-19T02:03:40.826 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7560999999999999e-05 2024-04-19T02:03:40.827 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.827 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.827 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.827 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.777136+0000", 2024-04-19T02:03:40.827 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000150023 2024-04-19T02:03:40.827 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.828 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.828 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.828 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.777161+0000", 2024-04-19T02:03:40.828 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5541000000000002e-05 2024-04-19T02:03:40.828 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.828 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.828 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.829 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.780357+0000", 2024-04-19T02:03:40.829 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0031956770000000001 2024-04-19T02:03:40.829 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.829 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.829 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.829 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.780453+0000", 2024-04-19T02:03:40.830 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5717000000000005e-05 2024-04-19T02:03:40.830 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.830 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.830 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.830 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.830 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.831 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.8s6 46/26 ECSubWrite(tid=81, reqid=osd.4.0:81, at_version=46'42, trim_to=0'0, roll_forward_to=45'40))", 2024-04-19T02:03:40.831 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.061957+0000", 2024-04-19T02:03:40.831 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 10.732301463000001, 2024-04-19T02:03:40.831 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0094780439999999997, 2024-04-19T02:03:40.831 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.831 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.832 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.832 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.832 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.832 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.832 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.061957+0000", 2024-04-19T02:03:40.832 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.832 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.833 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.833 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.833 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.061957+0000", 2024-04-19T02:03:40.833 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.833 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.833 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.833 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.833 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.061957+0000", 2024-04-19T02:03:40.834 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4700000000000001e-07 2024-04-19T02:03:40.834 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.834 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.834 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.834 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.062014+0000", 2024-04-19T02:03:40.834 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6895999999999999e-05 2024-04-19T02:03:40.834 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.835 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.835 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.835 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.062048+0000", 2024-04-19T02:03:40.835 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3597999999999999e-05 2024-04-19T02:03:40.835 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.835 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.835 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.835 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.062072+0000", 2024-04-19T02:03:40.835 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4204999999999999e-05 2024-04-19T02:03:40.836 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.836 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.836 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.836 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.062301+0000", 2024-04-19T02:03:40.837 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00022887999999999999 2024-04-19T02:03:40.837 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.837 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.837 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.837 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.062328+0000", 2024-04-19T02:03:40.837 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7280999999999999e-05 2024-04-19T02:03:40.837 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.837 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.837 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.838 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.071333+0000", 2024-04-19T02:03:40.838 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0090052770000000008 2024-04-19T02:03:40.838 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.838 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.838 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.838 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.071435+0000", 2024-04-19T02:03:40.838 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010116 2024-04-19T02:03:40.838 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.838 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.838 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.842 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.842 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.842 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.7s1 46/26 ECSubWrite(tid=158, reqid=osd.3.0:158, at_version=46'14, trim_to=0'0, roll_forward_to=45'12))", 2024-04-19T02:03:40.842 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.072342+0000", 2024-04-19T02:03:40.842 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 10.721916221000001, 2024-04-19T02:03:40.842 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0063685260000000002, 2024-04-19T02:03:40.842 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.842 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.842 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.842 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.842 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.843 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.843 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072342+0000", 2024-04-19T02:03:40.843 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.843 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.843 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.843 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.843 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072342+0000", 2024-04-19T02:03:40.843 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.843 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.843 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.843 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.843 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072343+0000", 2024-04-19T02:03:40.843 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9999999999999997e-07 2024-04-19T02:03:40.843 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.843 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.844 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.844 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072389+0000", 2024-04-19T02:03:40.844 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6427000000000001e-05 2024-04-19T02:03:40.844 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.844 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.844 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.844 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072416+0000", 2024-04-19T02:03:40.844 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7209e-05 2024-04-19T02:03:40.844 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.844 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.844 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.844 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072437+0000", 2024-04-19T02:03:40.844 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0874999999999999e-05 2024-04-19T02:03:40.844 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.844 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.845 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.845 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072649+0000", 2024-04-19T02:03:40.845 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000211821 2024-04-19T02:03:40.845 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.845 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.845 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.845 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072674+0000", 2024-04-19T02:03:40.845 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4734000000000001e-05 2024-04-19T02:03:40.845 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.845 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.845 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.845 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078610+0000", 2024-04-19T02:03:40.845 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0059367899999999999 2024-04-19T02:03:40.845 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.845 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.846 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.846 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078710+0000", 2024-04-19T02:03:40.846 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9970000000000007e-05 2024-04-19T02:03:40.846 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.846 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.846 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.846 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.846 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.846 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.bs3 46/26 ECSubWrite(tid=170, reqid=client.4509.0:442, at_version=46'29, trim_to=0'0, roll_forward_to=45'27))", 2024-04-19T02:03:40.846 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.218434+0000", 2024-04-19T02:03:40.846 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 10.575823643, 2024-04-19T02:03:40.846 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0040607229999999996, 2024-04-19T02:03:40.846 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.846 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.847 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.847 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.847 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.847 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.847 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.218434+0000", 2024-04-19T02:03:40.847 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.847 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.847 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.847 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.847 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.218434+0000", 2024-04-19T02:03:40.847 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.847 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.847 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.847 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.847 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.218437+0000", 2024-04-19T02:03:40.848 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3329999999999999e-06 2024-04-19T02:03:40.848 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.848 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.848 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.848 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.219290+0000", 2024-04-19T02:03:40.848 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00085335100000000002 2024-04-19T02:03:40.848 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.848 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.848 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.848 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.219325+0000", 2024-04-19T02:03:40.848 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4835000000000001e-05 2024-04-19T02:03:40.848 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.848 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.849 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.849 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.219343+0000", 2024-04-19T02:03:40.849 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8576e-05 2024-04-19T02:03:40.849 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.849 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.849 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.849 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.219541+0000", 2024-04-19T02:03:40.849 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00019705800000000001 2024-04-19T02:03:40.849 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.849 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.849 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.849 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.219568+0000", 2024-04-19T02:03:40.849 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7684000000000001e-05 2024-04-19T02:03:40.849 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.849 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.850 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.850 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.222419+0000", 2024-04-19T02:03:40.850 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0028505969999999999 2024-04-19T02:03:40.850 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.850 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.850 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.850 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.222495+0000", 2024-04-19T02:03:40.850 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6289000000000001e-05 2024-04-19T02:03:40.850 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.850 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.850 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.850 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.850 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.850 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.bs3 46/26 ECSubWrite(tid=177, reqid=client.4509.0:437, at_version=46'34, trim_to=0'0, roll_forward_to=46'32))", 2024-04-19T02:03:40.851 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.239790+0000", 2024-04-19T02:03:40.851 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 10.554468146, 2024-04-19T02:03:40.851 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0049614730000000001, 2024-04-19T02:03:40.851 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.851 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.851 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.851 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.851 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.851 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.851 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.239790+0000", 2024-04-19T02:03:40.851 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.851 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.851 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.851 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.852 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.239790+0000", 2024-04-19T02:03:40.852 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.852 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.852 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.852 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.852 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.239792+0000", 2024-04-19T02:03:40.852 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1730000000000002e-06 2024-04-19T02:03:40.852 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.852 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.852 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.852 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.240853+0000", 2024-04-19T02:03:40.852 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001060725 2024-04-19T02:03:40.852 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.852 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.852 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.853 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.240899+0000", 2024-04-19T02:03:40.853 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5933999999999998e-05 2024-04-19T02:03:40.853 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.853 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.853 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.853 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.240923+0000", 2024-04-19T02:03:40.853 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4426e-05 2024-04-19T02:03:40.853 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.853 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.853 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.853 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.241100+0000", 2024-04-19T02:03:40.853 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017696000000000001 2024-04-19T02:03:40.853 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.853 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.854 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.854 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.241127+0000", 2024-04-19T02:03:40.854 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7180999999999999e-05 2024-04-19T02:03:40.854 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.854 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.854 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.854 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.244633+0000", 2024-04-19T02:03:40.854 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0035052410000000001 2024-04-19T02:03:40.854 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.854 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.854 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.854 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.244751+0000", 2024-04-19T02:03:40.854 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011883300000000001 2024-04-19T02:03:40.855 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.855 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.855 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.855 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.855 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.855 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.bs3 47/26 ECSubWrite(tid=180, reqid=osd.3.0:180, at_version=47'36, trim_to=0'0, roll_forward_to=46'34))", 2024-04-19T02:03:40.855 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.081455+0000", 2024-04-19T02:03:40.855 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 9.7128031129999997, 2024-04-19T02:03:40.855 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0044551510000000001, 2024-04-19T02:03:40.855 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.855 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.855 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.855 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.855 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.855 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.856 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081455+0000", 2024-04-19T02:03:40.856 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.856 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.856 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.856 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.856 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081455+0000", 2024-04-19T02:03:40.856 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.856 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.856 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.856 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.856 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081456+0000", 2024-04-19T02:03:40.857 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3300000000000001e-07 2024-04-19T02:03:40.857 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.857 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.857 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.857 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081517+0000", 2024-04-19T02:03:40.857 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1080000000000005e-05 2024-04-19T02:03:40.857 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.857 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.857 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.857 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081562+0000", 2024-04-19T02:03:40.857 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5064000000000001e-05 2024-04-19T02:03:40.857 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.857 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.858 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.858 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081589+0000", 2024-04-19T02:03:40.858 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7605000000000001e-05 2024-04-19T02:03:40.858 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.858 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.858 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.858 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081850+0000", 2024-04-19T02:03:40.858 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00026099800000000002 2024-04-19T02:03:40.858 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.858 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.858 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.858 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081876+0000", 2024-04-19T02:03:40.858 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5488000000000001e-05 2024-04-19T02:03:40.858 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.858 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.859 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.859 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085816+0000", 2024-04-19T02:03:40.859 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0039403349999999997 2024-04-19T02:03:40.859 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.859 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.859 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.859 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085910+0000", 2024-04-19T02:03:40.859 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3848e-05 2024-04-19T02:03:40.859 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.859 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.859 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.859 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.859 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.859 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.bs3 47/26 ECSubWrite(tid=183, reqid=osd.3.0:183, at_version=47'38, trim_to=0'0, roll_forward_to=46'34))", 2024-04-19T02:03:40.860 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.081643+0000", 2024-04-19T02:03:40.860 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 9.7126146379999998, 2024-04-19T02:03:40.860 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0080822910000000001, 2024-04-19T02:03:40.860 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.860 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.860 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.860 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.860 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.860 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.860 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081643+0000", 2024-04-19T02:03:40.860 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.860 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.860 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.860 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.860 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081643+0000", 2024-04-19T02:03:40.861 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.861 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.861 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.861 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.861 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081644+0000", 2024-04-19T02:03:40.861 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8700000000000005e-07 2024-04-19T02:03:40.861 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.861 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.861 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.861 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081684+0000", 2024-04-19T02:03:40.861 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0271000000000001e-05 2024-04-19T02:03:40.861 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.861 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.861 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.861 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081706+0000", 2024-04-19T02:03:40.862 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1622000000000001e-05 2024-04-19T02:03:40.862 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.862 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.862 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.862 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081722+0000", 2024-04-19T02:03:40.862 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6322999999999999e-05 2024-04-19T02:03:40.862 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.862 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.862 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.862 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.083858+0000", 2024-04-19T02:03:40.862 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0021358979999999998 2024-04-19T02:03:40.862 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.862 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.862 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.863 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.083891+0000", 2024-04-19T02:03:40.863 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.2604999999999997e-05 2024-04-19T02:03:40.863 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.863 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.863 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.863 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.089628+0000", 2024-04-19T02:03:40.863 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0057371139999999998 2024-04-19T02:03:40.863 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.863 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.863 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.863 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.089726+0000", 2024-04-19T02:03:40.863 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7670999999999994e-05 2024-04-19T02:03:40.863 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.863 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.863 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.864 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.864 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.864 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.8s6 47/26 ECSubWrite(tid=86, reqid=osd.4.0:86, at_version=47'44, trim_to=0'0, roll_forward_to=46'42))", 2024-04-19T02:03:40.864 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.091163+0000", 2024-04-19T02:03:40.864 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 9.7030952629999998, 2024-04-19T02:03:40.864 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.005003281, 2024-04-19T02:03:40.864 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.864 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.864 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.864 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.864 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.864 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.864 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.091163+0000", 2024-04-19T02:03:40.865 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.865 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.865 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.865 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.865 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.091163+0000", 2024-04-19T02:03:40.865 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.865 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.865 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.865 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.865 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.091163+0000", 2024-04-19T02:03:40.865 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5700000000000002e-07 2024-04-19T02:03:40.865 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.865 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.865 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.865 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.091212+0000", 2024-04-19T02:03:40.865 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8269999999999997e-05 2024-04-19T02:03:40.866 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.866 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.866 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.866 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.091252+0000", 2024-04-19T02:03:40.866 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0515000000000001e-05 2024-04-19T02:03:40.866 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.866 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.866 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.866 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.091274+0000", 2024-04-19T02:03:40.866 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2130999999999999e-05 2024-04-19T02:03:40.866 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.866 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.866 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.866 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.091476+0000", 2024-04-19T02:03:40.867 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00020131200000000001 2024-04-19T02:03:40.867 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.867 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.867 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.867 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.091500+0000", 2024-04-19T02:03:40.867 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4771999999999999e-05 2024-04-19T02:03:40.867 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.867 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.867 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.867 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.096029+0000", 2024-04-19T02:03:40.867 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0045288459999999996 2024-04-19T02:03:40.867 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.867 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.867 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.868 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.096166+0000", 2024-04-19T02:03:40.868 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000136778 2024-04-19T02:03:40.868 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.868 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.868 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.868 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.868 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.868 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.8s6 47/26 ECSubWrite(tid=87, reqid=osd.4.0:86, at_version=0'0, trim_to=0'0, roll_forward_to=47'44))", 2024-04-19T02:03:40.868 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.118646+0000", 2024-04-19T02:03:40.868 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 9.6756115709999992, 2024-04-19T02:03:40.868 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0060891230000000001, 2024-04-19T02:03:40.868 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.868 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.868 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.868 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.869 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.869 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.869 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.118646+0000", 2024-04-19T02:03:40.869 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.869 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.869 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.869 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.869 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.118646+0000", 2024-04-19T02:03:40.869 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.869 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.869 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.869 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.869 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.118648+0000", 2024-04-19T02:03:40.869 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.843e-06 2024-04-19T02:03:40.869 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.870 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.870 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.870 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.118709+0000", 2024-04-19T02:03:40.870 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1147000000000003e-05 2024-04-19T02:03:40.870 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.870 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.870 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.870 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.118752+0000", 2024-04-19T02:03:40.870 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2997000000000002e-05 2024-04-19T02:03:40.870 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.870 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.870 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.870 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.118776+0000", 2024-04-19T02:03:40.870 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3189000000000001e-05 2024-04-19T02:03:40.871 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.871 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.871 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.871 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.119269+0000", 2024-04-19T02:03:40.871 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00049325300000000003 2024-04-19T02:03:40.871 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.871 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.871 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.871 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.119294+0000", 2024-04-19T02:03:40.871 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5381999999999999e-05 2024-04-19T02:03:40.871 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.871 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.871 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.871 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.124635+0000", 2024-04-19T02:03:40.871 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0053407120000000001 2024-04-19T02:03:40.872 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.872 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.872 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.872 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.124736+0000", 2024-04-19T02:03:40.872 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010060000000000001 2024-04-19T02:03:40.872 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.872 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.872 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.872 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.872 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.872 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.bs3 47/26 ECSubWrite(tid=186, reqid=osd.3.0:183, at_version=0'0, trim_to=0'0, roll_forward_to=47'38))", 2024-04-19T02:03:40.872 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.120984+0000", 2024-04-19T02:03:40.872 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 9.6732745399999995, 2024-04-19T02:03:40.872 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0064153229999999997, 2024-04-19T02:03:40.872 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.873 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.873 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.873 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.873 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.873 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.873 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.120984+0000", 2024-04-19T02:03:40.873 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.873 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.873 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.873 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.873 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.120984+0000", 2024-04-19T02:03:40.873 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.873 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.873 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.873 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.874 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.120985+0000", 2024-04-19T02:03:40.874 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9460000000000001e-06 2024-04-19T02:03:40.874 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.874 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.874 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.874 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.121014+0000", 2024-04-19T02:03:40.874 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8648000000000001e-05 2024-04-19T02:03:40.874 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.874 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.874 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.874 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.121049+0000", 2024-04-19T02:03:40.874 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4912e-05 2024-04-19T02:03:40.874 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.874 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.875 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.875 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.121072+0000", 2024-04-19T02:03:40.875 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3095000000000002e-05 2024-04-19T02:03:40.875 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.875 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.875 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.875 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.121286+0000", 2024-04-19T02:03:40.875 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00021347400000000001 2024-04-19T02:03:40.875 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.875 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.875 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.875 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.121311+0000", 2024-04-19T02:03:40.875 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5820999999999999e-05 2024-04-19T02:03:40.876 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.876 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.876 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.876 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.127303+0000", 2024-04-19T02:03:40.876 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0059911920000000002 2024-04-19T02:03:40.876 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.876 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.876 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.876 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.127399+0000", 2024-04-19T02:03:40.876 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6235000000000003e-05 2024-04-19T02:03:40.877 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.877 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.877 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.877 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.877 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.877 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.8s6 56/26 ECSubWrite(tid=110, reqid=client.4509.0:576, at_version=56'57, trim_to=0'0, roll_forward_to=55'55))", 2024-04-19T02:03:40.877 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:40.168846+0000", 2024-04-19T02:03:40.877 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.62541252199999997, 2024-04-19T02:03:40.877 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0055459899999999998, 2024-04-19T02:03:40.877 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.877 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.877 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.877 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.877 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.877 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.878 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.168846+0000", 2024-04-19T02:03:40.878 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.878 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.878 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.878 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.878 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.168846+0000", 2024-04-19T02:03:40.878 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.878 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.878 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.878 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.878 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.168846+0000", 2024-04-19T02:03:40.878 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7e-07 2024-04-19T02:03:40.878 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.878 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.879 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.879 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.169024+0000", 2024-04-19T02:03:40.879 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017762 2024-04-19T02:03:40.879 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.879 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.879 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.879 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.169049+0000", 2024-04-19T02:03:40.879 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5239e-05 2024-04-19T02:03:40.879 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.879 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.879 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.879 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.169061+0000", 2024-04-19T02:03:40.879 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2187e-05 2024-04-19T02:03:40.879 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.879 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.880 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.880 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.169903+0000", 2024-04-19T02:03:40.880 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00084208099999999995 2024-04-19T02:03:40.880 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.880 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.880 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.880 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.169929+0000", 2024-04-19T02:03:40.880 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6086000000000001e-05 2024-04-19T02:03:40.880 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.880 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.880 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.880 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.174340+0000", 2024-04-19T02:03:40.880 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0044111869999999996 2024-04-19T02:03:40.880 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.880 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.880 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.881 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.174392+0000", 2024-04-19T02:03:40.881 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1220000000000001e-05 2024-04-19T02:03:40.881 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.881 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.881 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.881 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.881 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.881 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.8s6 56/26 ECSubWrite(tid=112, reqid=client.4509.0:578, at_version=56'59, trim_to=0'0, roll_forward_to=55'55))", 2024-04-19T02:03:40.881 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:40.178180+0000", 2024-04-19T02:03:40.881 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.61607798199999997, 2024-04-19T02:03:40.881 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0042597590000000001, 2024-04-19T02:03:40.881 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.881 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.881 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.882 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.882 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.882 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.882 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.178180+0000", 2024-04-19T02:03:40.882 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.882 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.882 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.882 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.882 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.178180+0000", 2024-04-19T02:03:40.882 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.882 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.882 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.882 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.882 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.178181+0000", 2024-04-19T02:03:40.882 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.116e-06 2024-04-19T02:03:40.883 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.883 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.883 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.883 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.178312+0000", 2024-04-19T02:03:40.883 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000130665 2024-04-19T02:03:40.883 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.883 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.883 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.883 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.178335+0000", 2024-04-19T02:03:40.883 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3430000000000001e-05 2024-04-19T02:03:40.883 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.884 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.884 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.884 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.178362+0000", 2024-04-19T02:03:40.884 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6461e-05 2024-04-19T02:03:40.884 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.884 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.884 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.884 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.178493+0000", 2024-04-19T02:03:40.884 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013092600000000001 2024-04-19T02:03:40.884 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.884 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.884 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.884 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.178505+0000", 2024-04-19T02:03:40.884 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.272e-05 2024-04-19T02:03:40.884 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.885 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.885 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.885 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.182397+0000", 2024-04-19T02:03:40.885 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0038919480000000001 2024-04-19T02:03:40.885 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.885 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.885 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.885 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.182440+0000", 2024-04-19T02:03:40.885 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2493000000000001e-05 2024-04-19T02:03:40.885 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.885 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.885 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.885 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.885 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.885 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.8s6 56/26 ECSubWrite(tid=113, reqid=client.4509.0:579, at_version=56'60, trim_to=0'0, roll_forward_to=55'55))", 2024-04-19T02:03:40.886 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:40.180912+0000", 2024-04-19T02:03:40.886 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.613346328, 2024-04-19T02:03:40.886 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0041024879999999996, 2024-04-19T02:03:40.886 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.886 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.886 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.886 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.886 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.886 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.886 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.180912+0000", 2024-04-19T02:03:40.886 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.886 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.886 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.886 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.887 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.180912+0000", 2024-04-19T02:03:40.887 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.887 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.887 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.887 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.887 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.180913+0000", 2024-04-19T02:03:40.887 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0380000000000001e-06 2024-04-19T02:03:40.887 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.887 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.887 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.887 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.180968+0000", 2024-04-19T02:03:40.887 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5000000000000002e-05 2024-04-19T02:03:40.887 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.887 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.887 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.888 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.180985+0000", 2024-04-19T02:03:40.888 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7711e-05 2024-04-19T02:03:40.888 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.888 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.888 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.888 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.180996+0000", 2024-04-19T02:03:40.888 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0677e-05 2024-04-19T02:03:40.888 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.888 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.888 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.888 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.181132+0000", 2024-04-19T02:03:40.888 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000136169 2024-04-19T02:03:40.888 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.888 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.888 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.889 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.181144+0000", 2024-04-19T02:03:40.889 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2136999999999999e-05 2024-04-19T02:03:40.889 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.889 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.889 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.889 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.184927+0000", 2024-04-19T02:03:40.889 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0037824709999999999 2024-04-19T02:03:40.889 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.889 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.889 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.889 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.185014+0000", 2024-04-19T02:03:40.889 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7285000000000002e-05 2024-04-19T02:03:40.889 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.890 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.890 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.890 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.890 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.890 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.8s6 56/26 ECSubWrite(tid=114, reqid=client.4509.0:580, at_version=56'61, trim_to=0'0, roll_forward_to=56'57))", 2024-04-19T02:03:40.890 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:40.181113+0000", 2024-04-19T02:03:40.890 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.61314525600000003, 2024-04-19T02:03:40.890 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0042982289999999998, 2024-04-19T02:03:40.890 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:40.890 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:40.890 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:40.890 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:40.890 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.890 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:40.890 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.181113+0000", 2024-04-19T02:03:40.891 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.891 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.891 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.891 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:40.891 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.181113+0000", 2024-04-19T02:03:40.891 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:40.891 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.891 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.891 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:40.891 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.181114+0000", 2024-04-19T02:03:40.891 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3480000000000001e-06 2024-04-19T02:03:40.891 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.891 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.891 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:40.892 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.181169+0000", 2024-04-19T02:03:40.892 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4678000000000001e-05 2024-04-19T02:03:40.892 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.892 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.892 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:40.892 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.181196+0000", 2024-04-19T02:03:40.892 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7229999999999998e-05 2024-04-19T02:03:40.892 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.892 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.892 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:40.892 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.181216+0000", 2024-04-19T02:03:40.892 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9525e-05 2024-04-19T02:03:40.892 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.892 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.892 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:40.893 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.182130+0000", 2024-04-19T02:03:40.893 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00091421700000000005 2024-04-19T02:03:40.893 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.893 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.893 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:40.893 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.182142+0000", 2024-04-19T02:03:40.893 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1983e-05 2024-04-19T02:03:40.893 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.893 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.893 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:40.893 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.185367+0000", 2024-04-19T02:03:40.893 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0032254699999999998 2024-04-19T02:03:40.893 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:40.893 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:40.893 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:40.894 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.185411+0000", 2024-04-19T02:03:40.894 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3778e-05 2024-04-19T02:03:40.894 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.894 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.894 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.894 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:40.894 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:40.894 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:40.896 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-19T02:03:40.896 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:40.898+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:40.997 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:40.998+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:41.017 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:41.017 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:41.017 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 0 2024-04-19T02:03:41.018 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:41.030 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-19T02:03:41.078 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-19T02:03:41.098 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:41.094+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:41.140 INFO:tasks.rados.rados.0.smithi059.stdout:234: done (3 left) 2024-04-19T02:03:41.140 INFO:tasks.rados.rados.0.smithi059.stdout:235: snap_create 2024-04-19T02:03:41.140 INFO:tasks.rados.rados.0.smithi059.stdout:228: finishing write tid 5 to smithi05930959-46 2024-04-19T02:03:41.140 INFO:tasks.rados.rados.0.smithi059.stdout:228: oid 46 updating version 0 to 19 2024-04-19T02:03:41.141 INFO:tasks.rados.rados.0.smithi059.stdout:228: oid 46 updating version 19 to 22 2024-04-19T02:03:41.141 INFO:tasks.rados.rados.0.smithi059.stdout:228: oid 46 version 22 is already newer than 21 2024-04-19T02:03:41.141 INFO:tasks.rados.rados.0.smithi059.stdout:228: oid 46 version 22 is already newer than 20 2024-04-19T02:03:41.141 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 46 v 22 (ObjNum 109 snap 15 seq_num 109) dirty exists 2024-04-19T02:03:41.141 INFO:tasks.rados.rados.0.smithi059.stdout:228: left oid 46 (ObjNum 109 snap 15 seq_num 109) 2024-04-19T02:03:41.141 INFO:tasks.rados.rados.0.smithi059.stdout:232: finishing rollback tid 0 to smithi05930959-24 2024-04-19T02:03:41.141 INFO:tasks.rados.rados.0.smithi059.stdout:232: finishing rollback tid 1 to smithi05930959-24 2024-04-19T02:03:41.141 INFO:tasks.rados.rados.0.smithi059.stdout:231: expect (ObjNum 93 snap 9 seq_num 93) 2024-04-19T02:03:41.166 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:41.166 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:41.166 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-04-19T02:03:41.166 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-04-19T02:03:41.167 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:41.181 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-19T02:03:41.199 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:41.198+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:41.212 INFO:tasks.rados.rados.0.smithi059.stdout:232: finishing rollback tid 2 to smithi05930959-24 2024-04-19T02:03:41.212 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 24 v 18 (ObjNum 85 snap 8 seq_num 85) dirty exists 2024-04-19T02:03:41.292 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T02:03:41.339 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:41.342+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:41.345 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:41.345 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-04-19T02:03:41.345 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-04-19T02:03:41.345 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-04-19T02:03:41.345 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.345 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:351 3.es0 3:7af77758:::smithi05930959-34:head [append 0~622592] snapc 8=[8,6,5] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:41.345 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:24.760252+0000", 2024-04-19T02:03:41.345 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 16.584485816000001, 2024-04-19T02:03:41.345 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013544021999999999, 2024-04-19T02:03:41.345 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:41.346 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:41.346 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:41.346 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:41.346 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:41.346 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:41.346 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 351 2024-04-19T02:03:41.346 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.346 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:41.346 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.346 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:41.346 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.760252+0000", 2024-04-19T02:03:41.346 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.346 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.346 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.347 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:41.347 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.760252+0000", 2024-04-19T02:03:41.347 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.347 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.347 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.347 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:41.347 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.760255+0000", 2024-04-19T02:03:41.347 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.537e-06 2024-04-19T02:03:41.347 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.347 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.347 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:41.347 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.761579+0000", 2024-04-19T02:03:41.347 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0013242519999999999 2024-04-19T02:03:41.348 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.348 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.348 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:41.348 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.761611+0000", 2024-04-19T02:03:41.348 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1823999999999999e-05 2024-04-19T02:03:41.348 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.348 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.348 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:41.348 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.761637+0000", 2024-04-19T02:03:41.348 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6410999999999998e-05 2024-04-19T02:03:41.348 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.348 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.348 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:41.348 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.761780+0000", 2024-04-19T02:03:41.348 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014312 2024-04-19T02:03:41.349 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.349 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.349 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:41.349 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.761983+0000", 2024-04-19T02:03:41.349 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00020247599999999999 2024-04-19T02:03:41.349 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.349 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.349 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:41.349 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.765031+0000", 2024-04-19T02:03:41.349 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0030481140000000002 2024-04-19T02:03:41.349 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.349 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.349 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:41.349 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.768154+0000", 2024-04-19T02:03:41.349 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003123743 2024-04-19T02:03:41.350 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.350 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.350 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:41.350 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.773547+0000", 2024-04-19T02:03:41.350 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0053928099999999996 2024-04-19T02:03:41.350 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.350 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.350 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:41.350 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.773796+0000", 2024-04-19T02:03:41.350 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00024873499999999998 2024-04-19T02:03:41.350 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.350 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:41.350 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.350 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.351 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.351 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:352 3.es0 3:7af77758:::smithi05930959-34:head [append 0~475136] snapc 8=[8,6,5] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:41.351 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:24.767397+0000", 2024-04-19T02:03:41.351 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 16.577340958000001, 2024-04-19T02:03:41.351 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013273591, 2024-04-19T02:03:41.351 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:41.351 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:41.351 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:41.351 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:41.351 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:41.351 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:41.351 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 352 2024-04-19T02:03:41.351 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.351 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:41.351 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.352 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:41.352 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.767397+0000", 2024-04-19T02:03:41.352 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.352 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.352 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.352 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:41.352 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.767397+0000", 2024-04-19T02:03:41.352 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.352 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.352 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.352 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:41.352 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.767400+0000", 2024-04-19T02:03:41.352 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1530000000000001e-06 2024-04-19T02:03:41.352 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.352 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.353 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:41.353 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.768322+0000", 2024-04-19T02:03:41.353 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00092243399999999997 2024-04-19T02:03:41.353 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.353 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.353 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:41.353 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.768363+0000", 2024-04-19T02:03:41.353 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0657000000000001e-05 2024-04-19T02:03:41.353 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.353 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.353 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:41.353 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.768388+0000", 2024-04-19T02:03:41.353 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5128000000000001e-05 2024-04-19T02:03:41.353 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.353 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.353 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:41.354 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.768539+0000", 2024-04-19T02:03:41.354 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000150376 2024-04-19T02:03:41.354 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.354 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.354 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:41.354 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.768820+0000", 2024-04-19T02:03:41.354 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00028157699999999998 2024-04-19T02:03:41.354 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.354 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.354 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:41.354 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.771375+0000", 2024-04-19T02:03:41.354 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0025545730000000001 2024-04-19T02:03:41.354 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.354 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.354 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:41.355 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.779788+0000", 2024-04-19T02:03:41.355 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0084135930000000005 2024-04-19T02:03:41.355 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.355 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.355 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:41.355 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.780478+0000", 2024-04-19T02:03:41.355 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00068924100000000005 2024-04-19T02:03:41.355 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.355 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.355 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:41.355 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.780670+0000", 2024-04-19T02:03:41.355 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000192859 2024-04-19T02:03:41.355 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.355 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:41.355 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.356 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.356 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.356 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:355 3.es0 3:7af77758:::smithi05930959-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:41.356 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:24.772697+0000", 2024-04-19T02:03:41.356 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 16.572040555000001, 2024-04-19T02:03:41.356 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014504697, 2024-04-19T02:03:41.356 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:41.356 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:41.356 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:03:41.356 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:41.356 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:41.356 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:41.356 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 355 2024-04-19T02:03:41.356 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.356 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:41.356 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.357 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:41.357 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.772697+0000", 2024-04-19T02:03:41.357 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.357 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.357 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.357 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:41.357 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.772697+0000", 2024-04-19T02:03:41.357 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.357 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.357 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.357 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:41.357 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.772699+0000", 2024-04-19T02:03:41.357 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3430000000000001e-06 2024-04-19T02:03:41.357 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.357 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.357 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:41.358 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.772706+0000", 2024-04-19T02:03:41.358 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2400000000000001e-06 2024-04-19T02:03:41.358 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.358 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.358 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:41.358 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.772724+0000", 2024-04-19T02:03:41.358 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8289000000000001e-05 2024-04-19T02:03:41.358 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.358 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.358 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:41.358 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.772738+0000", 2024-04-19T02:03:41.358 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3732e-05 2024-04-19T02:03:41.358 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.359 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.359 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:41.359 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.784149+0000", 2024-04-19T02:03:41.359 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011411361 2024-04-19T02:03:41.359 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.359 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.359 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:41.359 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.784288+0000", 2024-04-19T02:03:41.359 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013880699999999999 2024-04-19T02:03:41.359 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.359 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.359 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:41.359 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.787202+0000", 2024-04-19T02:03:41.360 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0029139249999999999 2024-04-19T02:03:41.360 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.360 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:41.360 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.360 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.360 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.360 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.bs7 42/26 ECSubWrite(tid=133, reqid=osd.3.0:133, at_version=42'21, trim_to=0'0, roll_forward_to=39'19))", 2024-04-19T02:03:41.360 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.748986+0000", 2024-04-19T02:03:41.360 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 14.595751385, 2024-04-19T02:03:41.360 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.015850517000000001, 2024-04-19T02:03:41.360 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:41.360 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:41.360 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:41.360 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:41.360 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.361 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:41.361 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748986+0000", 2024-04-19T02:03:41.361 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.361 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.361 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.361 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:41.361 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748986+0000", 2024-04-19T02:03:41.361 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.361 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.361 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.361 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:41.361 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748987+0000", 2024-04-19T02:03:41.361 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6000000000000003e-07 2024-04-19T02:03:41.361 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.362 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.362 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:41.362 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749579+0000", 2024-04-19T02:03:41.362 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00059223999999999995 2024-04-19T02:03:41.362 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.362 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.362 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:41.362 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749611+0000", 2024-04-19T02:03:41.362 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1554999999999999e-05 2024-04-19T02:03:41.362 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.362 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.362 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:41.362 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749633+0000", 2024-04-19T02:03:41.362 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1885000000000001e-05 2024-04-19T02:03:41.363 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.363 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.363 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:41.363 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749799+0000", 2024-04-19T02:03:41.363 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000166422 2024-04-19T02:03:41.363 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.363 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.363 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:41.363 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749825+0000", 2024-04-19T02:03:41.363 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5825000000000001e-05 2024-04-19T02:03:41.363 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.363 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.363 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:41.363 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.764686+0000", 2024-04-19T02:03:41.363 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014861087 2024-04-19T02:03:41.363 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.364 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.364 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:41.364 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.764837+0000", 2024-04-19T02:03:41.364 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000150743 2024-04-19T02:03:41.364 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.364 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:41.364 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.364 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.364 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.364 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.bs7 42/26 ECSubWrite(tid=135, reqid=osd.3.0:135, at_version=42'23, trim_to=0'0, roll_forward_to=39'19))", 2024-04-19T02:03:41.364 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.749928+0000", 2024-04-19T02:03:41.364 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 14.594810001000001, 2024-04-19T02:03:41.364 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.017644765, 2024-04-19T02:03:41.365 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:41.365 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:41.365 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:41.365 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:41.365 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.365 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:41.365 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749928+0000", 2024-04-19T02:03:41.365 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.365 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.365 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.365 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:41.365 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749928+0000", 2024-04-19T02:03:41.365 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.365 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.365 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.366 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:41.366 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749929+0000", 2024-04-19T02:03:41.366 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.136e-06 2024-04-19T02:03:41.366 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.366 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.366 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:41.366 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749999+0000", 2024-04-19T02:03:41.366 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0471999999999999e-05 2024-04-19T02:03:41.366 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.366 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.366 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:41.366 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.750024+0000", 2024-04-19T02:03:41.366 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4575000000000001e-05 2024-04-19T02:03:41.366 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.366 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.367 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:41.367 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.750044+0000", 2024-04-19T02:03:41.367 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9802000000000001e-05 2024-04-19T02:03:41.367 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.367 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.367 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:41.367 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.756068+0000", 2024-04-19T02:03:41.367 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0060238050000000001 2024-04-19T02:03:41.367 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.367 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.367 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:41.367 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.756097+0000", 2024-04-19T02:03:41.367 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9045000000000001e-05 2024-04-19T02:03:41.367 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.368 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.368 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:41.368 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.767487+0000", 2024-04-19T02:03:41.368 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011390612 2024-04-19T02:03:41.368 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.368 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.368 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:41.368 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.767573+0000", 2024-04-19T02:03:41.368 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5317999999999999e-05 2024-04-19T02:03:41.368 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.368 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:41.368 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.368 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.368 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.368 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.4s4 42/26 ECSubWrite(tid=45, reqid=osd.1.0:44, at_version=0'0, trim_to=0'0, roll_forward_to=42'32))", 2024-04-19T02:03:41.369 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.755829+0000", 2024-04-19T02:03:41.369 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 14.588908931000001, 2024-04-19T02:03:41.369 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013143623, 2024-04-19T02:03:41.369 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:41.369 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:41.369 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:41.369 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:41.369 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.369 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:41.369 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755829+0000", 2024-04-19T02:03:41.369 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.369 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.369 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.369 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:41.369 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755829+0000", 2024-04-19T02:03:41.370 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.370 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.370 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.370 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:41.370 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755831+0000", 2024-04-19T02:03:41.370 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6929999999999999e-06 2024-04-19T02:03:41.370 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.370 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.370 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:41.370 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755858+0000", 2024-04-19T02:03:41.370 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7067999999999999e-05 2024-04-19T02:03:41.370 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.370 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.370 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:41.371 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755887+0000", 2024-04-19T02:03:41.371 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9651000000000001e-05 2024-04-19T02:03:41.371 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.371 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.371 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:41.371 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755910+0000", 2024-04-19T02:03:41.371 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2311999999999999e-05 2024-04-19T02:03:41.371 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.371 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.371 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:41.371 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.756066+0000", 2024-04-19T02:03:41.371 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000156176 2024-04-19T02:03:41.371 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.371 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.371 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:41.372 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.756092+0000", 2024-04-19T02:03:41.372 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5825000000000001e-05 2024-04-19T02:03:41.372 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.372 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.372 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:41.372 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.768920+0000", 2024-04-19T02:03:41.372 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012828104 2024-04-19T02:03:41.372 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.372 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.372 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:41.372 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.768972+0000", 2024-04-19T02:03:41.372 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2794000000000003e-05 2024-04-19T02:03:41.372 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.372 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:41.373 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.373 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.373 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.373 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.ds2 46/26 ECSubWrite(tid=187, reqid=osd.8.0:187, at_version=46'35, trim_to=0'0, roll_forward_to=45'31))", 2024-04-19T02:03:41.373 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.058580+0000", 2024-04-19T02:03:41.373 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 11.286157902999999, 2024-04-19T02:03:41.373 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.016482007999999999, 2024-04-19T02:03:41.373 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:41.373 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:41.373 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:41.373 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:41.373 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.373 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:41.373 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.058580+0000", 2024-04-19T02:03:41.373 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.373 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.374 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.374 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:41.374 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.058580+0000", 2024-04-19T02:03:41.374 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.374 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.374 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.374 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:41.374 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.058582+0000", 2024-04-19T02:03:41.374 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7069999999999999e-06 2024-04-19T02:03:41.374 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.374 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.374 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:41.374 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.058632+0000", 2024-04-19T02:03:41.374 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0710000000000001e-05 2024-04-19T02:03:41.375 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.375 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.375 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:41.375 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.058664+0000", 2024-04-19T02:03:41.375 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1448000000000001e-05 2024-04-19T02:03:41.375 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.375 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.375 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:41.375 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.058688+0000", 2024-04-19T02:03:41.375 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4405000000000001e-05 2024-04-19T02:03:41.375 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.375 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.375 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:41.375 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.059171+0000", 2024-04-19T02:03:41.375 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00048236699999999998 2024-04-19T02:03:41.375 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.376 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.376 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:41.376 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.059195+0000", 2024-04-19T02:03:41.376 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4042000000000001e-05 2024-04-19T02:03:41.376 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.376 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.376 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:41.376 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.074940+0000", 2024-04-19T02:03:41.376 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.015745333 2024-04-19T02:03:41.376 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.376 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.376 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:41.376 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.075062+0000", 2024-04-19T02:03:41.376 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000121996 2024-04-19T02:03:41.376 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.377 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:41.377 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.377 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.377 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.377 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.4s4 46/26 ECSubWrite(tid=58, reqid=osd.1.0:58, at_version=46'44, trim_to=0'0, roll_forward_to=45'42))", 2024-04-19T02:03:41.377 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.059794+0000", 2024-04-19T02:03:41.377 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 11.284944224, 2024-04-19T02:03:41.377 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.015185597, 2024-04-19T02:03:41.377 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:41.377 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:41.377 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:41.377 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:41.377 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.377 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:41.378 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.059794+0000", 2024-04-19T02:03:41.378 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.378 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.378 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.378 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:41.378 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.059794+0000", 2024-04-19T02:03:41.378 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.378 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.378 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.378 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:41.378 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.059794+0000", 2024-04-19T02:03:41.378 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2399999999999998e-07 2024-04-19T02:03:41.378 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.378 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.378 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:41.379 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.059843+0000", 2024-04-19T02:03:41.379 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9217e-05 2024-04-19T02:03:41.379 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.379 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.379 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:41.379 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.059888+0000", 2024-04-19T02:03:41.379 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4842999999999997e-05 2024-04-19T02:03:41.379 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.379 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.379 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:41.379 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.059911+0000", 2024-04-19T02:03:41.379 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2829000000000001e-05 2024-04-19T02:03:41.379 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.379 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.379 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:41.380 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.060084+0000", 2024-04-19T02:03:41.380 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017278000000000001 2024-04-19T02:03:41.380 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.380 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.380 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:41.380 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.060110+0000", 2024-04-19T02:03:41.380 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5931999999999998e-05 2024-04-19T02:03:41.380 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.380 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.380 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:41.380 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.074866+0000", 2024-04-19T02:03:41.380 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014756136 2024-04-19T02:03:41.380 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.380 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.380 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:41.381 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.074979+0000", 2024-04-19T02:03:41.381 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000113236 2024-04-19T02:03:41.381 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.381 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:41.381 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.381 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.381 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.381 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.as5 46/26 ECSubWrite(tid=102, reqid=osd.11.0:102, at_version=46'47, trim_to=0'0, roll_forward_to=45'45))", 2024-04-19T02:03:41.381 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.066484+0000", 2024-04-19T02:03:41.381 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 11.278254240000001, 2024-04-19T02:03:41.381 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014065139000000001, 2024-04-19T02:03:41.381 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:41.381 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:41.381 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:41.381 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:41.382 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.382 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:41.382 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066484+0000", 2024-04-19T02:03:41.382 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.382 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.382 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.382 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:41.382 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066484+0000", 2024-04-19T02:03:41.382 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.382 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.382 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.382 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:41.382 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066484+0000", 2024-04-19T02:03:41.382 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0299999999999998e-07 2024-04-19T02:03:41.383 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.383 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.383 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:41.383 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066542+0000", 2024-04-19T02:03:41.383 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7182999999999998e-05 2024-04-19T02:03:41.383 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.383 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.383 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:41.383 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066572+0000", 2024-04-19T02:03:41.383 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0205e-05 2024-04-19T02:03:41.383 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.383 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.383 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:41.383 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066594+0000", 2024-04-19T02:03:41.383 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2478999999999999e-05 2024-04-19T02:03:41.383 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.384 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.384 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:41.384 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066759+0000", 2024-04-19T02:03:41.384 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016516999999999999 2024-04-19T02:03:41.384 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.384 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.384 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:41.384 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066784+0000", 2024-04-19T02:03:41.384 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4587999999999999e-05 2024-04-19T02:03:41.384 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.384 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.384 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:41.384 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.080462+0000", 2024-04-19T02:03:41.384 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013678459 2024-04-19T02:03:41.384 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.385 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.385 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:41.385 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.080549+0000", 2024-04-19T02:03:41.385 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.6252e-05 2024-04-19T02:03:41.385 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.385 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:41.385 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.385 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.385 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.385 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.cs3 46/26 ECSubWrite(tid=44, reqid=osd.9.0:44, at_version=46'26, trim_to=0'0, roll_forward_to=45'24))", 2024-04-19T02:03:41.385 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.066755+0000", 2024-04-19T02:03:41.385 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 11.277982473, 2024-04-19T02:03:41.385 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.01420738, 2024-04-19T02:03:41.385 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:41.386 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:41.386 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:41.386 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:41.386 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.386 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:41.386 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066755+0000", 2024-04-19T02:03:41.386 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.386 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.386 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.386 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:41.386 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066755+0000", 2024-04-19T02:03:41.386 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.386 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.386 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.386 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:41.387 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066756+0000", 2024-04-19T02:03:41.387 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4000000000000002e-07 2024-04-19T02:03:41.387 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.387 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.387 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:41.387 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066799+0000", 2024-04-19T02:03:41.387 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3217000000000003e-05 2024-04-19T02:03:41.387 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.387 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.387 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:41.387 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066825+0000", 2024-04-19T02:03:41.387 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6248999999999999e-05 2024-04-19T02:03:41.387 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.387 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.388 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:41.388 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066851+0000", 2024-04-19T02:03:41.388 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5171e-05 2024-04-19T02:03:41.388 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.388 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.388 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:41.388 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.067016+0000", 2024-04-19T02:03:41.388 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016518699999999999 2024-04-19T02:03:41.388 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.388 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.388 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:41.388 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.067042+0000", 2024-04-19T02:03:41.388 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5992000000000001e-05 2024-04-19T02:03:41.388 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.388 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.388 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:41.389 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.080884+0000", 2024-04-19T02:03:41.389 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013842047999999999 2024-04-19T02:03:41.389 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.389 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.389 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:41.389 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.080963+0000", 2024-04-19T02:03:41.389 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8975999999999997e-05 2024-04-19T02:03:41.389 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.389 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:41.389 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.389 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.389 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.389 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.as5 46/26 ECSubWrite(tid=103, reqid=osd.11.0:103, at_version=46'49, trim_to=0'0, roll_forward_to=45'45))", 2024-04-19T02:03:41.389 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.067427+0000", 2024-04-19T02:03:41.390 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 11.277310308000001, 2024-04-19T02:03:41.390 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.020157152000000001, 2024-04-19T02:03:41.390 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:41.390 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:41.390 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:41.390 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:41.390 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.390 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:41.390 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.067427+0000", 2024-04-19T02:03:41.390 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.390 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.390 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.390 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:41.390 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.067427+0000", 2024-04-19T02:03:41.390 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.391 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.391 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.391 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:41.391 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.067429+0000", 2024-04-19T02:03:41.391 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7439999999999999e-06 2024-04-19T02:03:41.391 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.391 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.391 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:41.391 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.067489+0000", 2024-04-19T02:03:41.391 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9988999999999998e-05 2024-04-19T02:03:41.391 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.391 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.391 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:41.391 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.068116+0000", 2024-04-19T02:03:41.391 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00062675199999999995 2024-04-19T02:03:41.392 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.392 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.392 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:41.392 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.068146+0000", 2024-04-19T02:03:41.392 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9751999999999999e-05 2024-04-19T02:03:41.392 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.392 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.392 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:41.392 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.070935+0000", 2024-04-19T02:03:41.392 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0027894399999999998 2024-04-19T02:03:41.392 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.392 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.392 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:41.392 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.070950+0000", 2024-04-19T02:03:41.392 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5218999999999999e-05 2024-04-19T02:03:41.393 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.393 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.393 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:41.393 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.087492+0000", 2024-04-19T02:03:41.393 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.016541891999999999 2024-04-19T02:03:41.393 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.393 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.393 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:41.393 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.087585+0000", 2024-04-19T02:03:41.393 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.2363999999999999e-05 2024-04-19T02:03:41.393 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.393 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:41.393 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.393 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.393 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.394 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.7s4 46/26 ECSubWrite(tid=158, reqid=osd.3.0:158, at_version=46'14, trim_to=0'0, roll_forward_to=45'12))", 2024-04-19T02:03:41.394 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.072853+0000", 2024-04-19T02:03:41.394 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 11.271884311000001, 2024-04-19T02:03:41.394 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014228316, 2024-04-19T02:03:41.394 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:41.394 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:41.394 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:41.394 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:41.394 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.394 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:41.394 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072853+0000", 2024-04-19T02:03:41.394 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.394 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.394 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.395 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:41.395 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072853+0000", 2024-04-19T02:03:41.395 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.395 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.395 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.395 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:41.395 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072854+0000", 2024-04-19T02:03:41.395 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5700000000000002e-07 2024-04-19T02:03:41.395 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.395 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.395 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:41.395 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072905+0000", 2024-04-19T02:03:41.395 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1443000000000002e-05 2024-04-19T02:03:41.395 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.395 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.396 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:41.396 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072931+0000", 2024-04-19T02:03:41.396 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5103e-05 2024-04-19T02:03:41.396 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.396 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.396 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:41.396 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072953+0000", 2024-04-19T02:03:41.396 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2174000000000001e-05 2024-04-19T02:03:41.396 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.396 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.396 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:41.396 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.073226+0000", 2024-04-19T02:03:41.396 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000273174 2024-04-19T02:03:41.396 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.396 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.397 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:41.397 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.073252+0000", 2024-04-19T02:03:41.397 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6152e-05 2024-04-19T02:03:41.397 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.397 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.397 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:41.397 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.086987+0000", 2024-04-19T02:03:41.397 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013734837999999999 2024-04-19T02:03:41.397 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.397 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.397 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:41.397 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.087082+0000", 2024-04-19T02:03:41.397 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4875000000000002e-05 2024-04-19T02:03:41.397 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.397 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:41.398 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.398 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.398 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.398 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.4s4 46/26 ECSubWrite(tid=59, reqid=osd.1.0:58, at_version=0'0, trim_to=0'0, roll_forward_to=46'44))", 2024-04-19T02:03:41.398 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.076912+0000", 2024-04-19T02:03:41.398 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 11.26782539, 2024-04-19T02:03:41.398 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014831941, 2024-04-19T02:03:41.398 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:41.398 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:41.398 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:41.398 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:41.398 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.398 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:41.398 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.076912+0000", 2024-04-19T02:03:41.399 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.399 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.399 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.399 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:41.399 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.076912+0000", 2024-04-19T02:03:41.399 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.399 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.399 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.399 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:41.399 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.076914+0000", 2024-04-19T02:03:41.399 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.829e-06 2024-04-19T02:03:41.399 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.399 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.399 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:41.400 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.076944+0000", 2024-04-19T02:03:41.400 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9482e-05 2024-04-19T02:03:41.400 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.400 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.400 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:41.400 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.080072+0000", 2024-04-19T02:03:41.400 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0031280629999999999 2024-04-19T02:03:41.400 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.400 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.400 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:41.400 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.080101+0000", 2024-04-19T02:03:41.400 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9524999999999999e-05 2024-04-19T02:03:41.400 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.400 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.400 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:41.401 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.081114+0000", 2024-04-19T02:03:41.401 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0010130250000000001 2024-04-19T02:03:41.401 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.401 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.401 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:41.401 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.081138+0000", 2024-04-19T02:03:41.401 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4105e-05 2024-04-19T02:03:41.401 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.401 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.401 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:41.401 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.091652+0000", 2024-04-19T02:03:41.401 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010514055 2024-04-19T02:03:41.401 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.401 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.401 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:41.402 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.091744+0000", 2024-04-19T02:03:41.402 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1856999999999998e-05 2024-04-19T02:03:41.402 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.402 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:41.402 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.402 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.402 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.402 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.1s5 46/26 ECSubWrite(tid=21, reqid=client.4509.0:416, at_version=46'9, trim_to=0'0, roll_forward_to=29'6))", 2024-04-19T02:03:41.402 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.081553+0000", 2024-04-19T02:03:41.402 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 11.263185046, 2024-04-19T02:03:41.402 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013737042, 2024-04-19T02:03:41.402 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:41.402 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:41.402 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:41.403 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:41.403 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.403 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:41.403 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.081553+0000", 2024-04-19T02:03:41.403 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.403 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.403 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.403 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:41.403 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.081553+0000", 2024-04-19T02:03:41.403 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.403 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.403 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.403 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:41.403 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.081554+0000", 2024-04-19T02:03:41.403 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.083e-06 2024-04-19T02:03:41.404 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.404 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.404 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:41.404 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.081793+0000", 2024-04-19T02:03:41.404 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00023883199999999999 2024-04-19T02:03:41.404 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.404 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.404 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:41.404 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.081820+0000", 2024-04-19T02:03:41.404 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7429000000000001e-05 2024-04-19T02:03:41.404 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.404 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.404 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:41.404 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.081840+0000", 2024-04-19T02:03:41.404 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9508e-05 2024-04-19T02:03:41.405 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.405 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.405 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:41.405 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.085041+0000", 2024-04-19T02:03:41.405 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0032012859999999998 2024-04-19T02:03:41.405 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.405 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.405 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:41.405 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.085066+0000", 2024-04-19T02:03:41.405 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5222e-05 2024-04-19T02:03:41.405 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.405 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.405 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:41.405 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.095186+0000", 2024-04-19T02:03:41.405 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010120168000000001 2024-04-19T02:03:41.406 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.406 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.406 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:41.406 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.095290+0000", 2024-04-19T02:03:41.406 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000103514 2024-04-19T02:03:41.406 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.406 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:41.406 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.406 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.406 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.406 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:420 3.es0 3:7831c41d:::smithi05930959-30:head [copy-from ver 14] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:41.406 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.193498+0000", 2024-04-19T02:03:41.406 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 11.151239714000001, 2024-04-19T02:03:41.406 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.052261775000000003, 2024-04-19T02:03:41.407 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:41.407 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:41.407 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:41.407 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:41.407 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:41.407 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:41.407 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 420 2024-04-19T02:03:41.407 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.407 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:41.407 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.407 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:41.407 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193498+0000", 2024-04-19T02:03:41.407 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.407 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.407 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.408 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:41.408 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193498+0000", 2024-04-19T02:03:41.408 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.408 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.408 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.408 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:41.408 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193501+0000", 2024-04-19T02:03:41.408 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.88e-06 2024-04-19T02:03:41.408 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.408 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.408 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:41.408 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193514+0000", 2024-04-19T02:03:41.408 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2879e-05 2024-04-19T02:03:41.408 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.408 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.408 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:41.409 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193546+0000", 2024-04-19T02:03:41.409 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1931999999999999e-05 2024-04-19T02:03:41.409 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.409 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.409 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:41.409 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193560+0000", 2024-04-19T02:03:41.409 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4609e-05 2024-04-19T02:03:41.409 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.409 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.409 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:41.409 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193695+0000", 2024-04-19T02:03:41.409 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013434499999999999 2024-04-19T02:03:41.409 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.409 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.409 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:41.409 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193864+0000", 2024-04-19T02:03:41.410 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016894300000000001 2024-04-19T02:03:41.410 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.410 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.410 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:41.410 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.239884+0000", 2024-04-19T02:03:41.410 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.046020559000000003 2024-04-19T02:03:41.410 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.410 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.410 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:41.410 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.245339+0000", 2024-04-19T02:03:41.410 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0054546569999999999 2024-04-19T02:03:41.410 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.410 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.410 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:41.410 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.245619+0000", 2024-04-19T02:03:41.411 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00028016800000000002 2024-04-19T02:03:41.411 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.411 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.411 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:41.411 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.245760+0000", 2024-04-19T02:03:41.411 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000140803 2024-04-19T02:03:41.411 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.411 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:41.411 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.411 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.411 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.411 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:421 3.es0 3:7831c41d:::smithi05930959-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:41.411 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.193593+0000", 2024-04-19T02:03:41.411 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 11.151145209999999, 2024-04-19T02:03:41.411 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.049216688000000001, 2024-04-19T02:03:41.412 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:41.412 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:41.412 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:03:41.412 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:41.412 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:41.412 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:41.412 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 421 2024-04-19T02:03:41.412 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.412 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:41.412 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.412 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:41.412 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193593+0000", 2024-04-19T02:03:41.412 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.412 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.412 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.412 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:41.413 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193593+0000", 2024-04-19T02:03:41.413 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.413 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.413 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.413 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:41.413 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193594+0000", 2024-04-19T02:03:41.413 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4169999999999999e-06 2024-04-19T02:03:41.413 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.413 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.413 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:41.413 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193601+0000", 2024-04-19T02:03:41.413 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6429999999999998e-06 2024-04-19T02:03:41.413 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.413 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.414 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:41.414 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193621+0000", 2024-04-19T02:03:41.414 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0435e-05 2024-04-19T02:03:41.414 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.414 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.414 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:41.414 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193636+0000", 2024-04-19T02:03:41.414 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4822e-05 2024-04-19T02:03:41.414 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.414 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.414 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:41.414 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.194259+0000", 2024-04-19T02:03:41.414 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00062306299999999998 2024-04-19T02:03:41.414 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.414 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.415 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for blocked object", 2024-04-19T02:03:41.415 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.194433+0000", 2024-04-19T02:03:41.415 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017404899999999999 2024-04-19T02:03:41.415 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.415 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.415 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:41.415 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.242239+0000", 2024-04-19T02:03:41.415 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.047806081 2024-04-19T02:03:41.415 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.415 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.415 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:41.415 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.242500+0000", 2024-04-19T02:03:41.415 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00026063699999999998 2024-04-19T02:03:41.415 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.415 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.415 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:41.416 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.242809+0000", 2024-04-19T02:03:41.416 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00030954099999999998 2024-04-19T02:03:41.416 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.416 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:41.416 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.416 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.416 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.416 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.bs7 47/26 ECSubWrite(tid=183, reqid=osd.3.0:183, at_version=47'38, trim_to=0'0, roll_forward_to=46'34))", 2024-04-19T02:03:41.416 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.084378+0000", 2024-04-19T02:03:41.416 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 10.260360123, 2024-04-19T02:03:41.416 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.020336838999999999, 2024-04-19T02:03:41.416 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:41.416 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:41.416 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:41.416 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:41.416 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.417 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:41.417 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.084378+0000", 2024-04-19T02:03:41.417 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.417 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.417 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.417 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:41.417 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.084378+0000", 2024-04-19T02:03:41.417 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.417 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.417 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.417 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:41.417 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.084379+0000", 2024-04-19T02:03:41.417 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3230000000000001e-06 2024-04-19T02:03:41.417 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.417 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.418 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:41.418 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.084443+0000", 2024-04-19T02:03:41.418 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3745999999999996e-05 2024-04-19T02:03:41.418 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.418 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.418 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:41.418 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.084476+0000", 2024-04-19T02:03:41.418 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3352000000000002e-05 2024-04-19T02:03:41.418 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.418 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.418 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:41.418 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.084500+0000", 2024-04-19T02:03:41.418 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3492000000000001e-05 2024-04-19T02:03:41.418 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.419 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.419 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:41.419 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.084742+0000", 2024-04-19T02:03:41.419 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00024195900000000001 2024-04-19T02:03:41.419 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.419 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.419 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:41.419 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.084769+0000", 2024-04-19T02:03:41.419 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7222000000000001e-05 2024-04-19T02:03:41.419 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.419 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.419 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:41.419 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.104626+0000", 2024-04-19T02:03:41.419 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.019857593999999999 2024-04-19T02:03:41.419 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.419 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.420 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:41.420 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.104715+0000", 2024-04-19T02:03:41.420 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8151000000000004e-05 2024-04-19T02:03:41.420 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.420 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:41.420 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.420 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.420 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.420 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.1s5 47/26 ECSubWrite(tid=30, reqid=osd.0.0:30, at_version=47'17, trim_to=0'0, roll_forward_to=46'13))", 2024-04-19T02:03:41.420 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.085073+0000", 2024-04-19T02:03:41.420 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 10.259665097999999, 2024-04-19T02:03:41.420 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.019688054999999999, 2024-04-19T02:03:41.420 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:41.420 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:41.420 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:41.420 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:41.421 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.421 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:41.421 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085073+0000", 2024-04-19T02:03:41.421 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.421 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.421 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.421 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:41.421 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085073+0000", 2024-04-19T02:03:41.421 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.421 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.421 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.421 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:41.421 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085075+0000", 2024-04-19T02:03:41.421 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.81e-06 2024-04-19T02:03:41.421 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.422 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.422 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:41.422 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085126+0000", 2024-04-19T02:03:41.422 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1100000000000002e-05 2024-04-19T02:03:41.422 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.422 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.422 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:41.422 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085158+0000", 2024-04-19T02:03:41.422 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.2375000000000001e-05 2024-04-19T02:03:41.422 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.422 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.422 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:41.422 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085180+0000", 2024-04-19T02:03:41.422 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1942000000000001e-05 2024-04-19T02:03:41.422 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.423 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.423 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:41.423 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.086348+0000", 2024-04-19T02:03:41.423 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001167757 2024-04-19T02:03:41.423 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.423 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.423 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:41.423 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.086362+0000", 2024-04-19T02:03:41.423 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4749000000000001e-05 2024-04-19T02:03:41.423 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.423 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.423 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:41.423 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.104692+0000", 2024-04-19T02:03:41.423 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.018329525999999999 2024-04-19T02:03:41.423 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.423 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.424 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:41.424 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.104761+0000", 2024-04-19T02:03:41.424 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8795999999999997e-05 2024-04-19T02:03:41.424 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.424 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:41.424 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.424 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.424 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.424 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.8s2 47/26 ECSubWrite(tid=86, reqid=osd.4.0:86, at_version=47'44, trim_to=0'0, roll_forward_to=46'42))", 2024-04-19T02:03:41.424 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.093211+0000", 2024-04-19T02:03:41.424 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 10.251527039999999, 2024-04-19T02:03:41.424 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014295323, 2024-04-19T02:03:41.424 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:41.424 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:41.424 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:41.425 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:41.425 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.425 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:41.425 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.093211+0000", 2024-04-19T02:03:41.425 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.425 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.425 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.425 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:41.425 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.093211+0000", 2024-04-19T02:03:41.425 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.425 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.425 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.425 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:41.425 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.093211+0000", 2024-04-19T02:03:41.426 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3900000000000004e-07 2024-04-19T02:03:41.426 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.426 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.426 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:41.426 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.093266+0000", 2024-04-19T02:03:41.426 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4684e-05 2024-04-19T02:03:41.426 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.426 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.426 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:41.426 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.093292+0000", 2024-04-19T02:03:41.426 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6001999999999999e-05 2024-04-19T02:03:41.426 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.426 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.426 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:41.427 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.093308+0000", 2024-04-19T02:03:41.427 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6052000000000001e-05 2024-04-19T02:03:41.427 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.427 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.427 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:41.427 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.093541+0000", 2024-04-19T02:03:41.427 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00023266000000000001 2024-04-19T02:03:41.427 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.427 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.427 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:41.427 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.093573+0000", 2024-04-19T02:03:41.427 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1908e-05 2024-04-19T02:03:41.427 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.427 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.427 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:41.428 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.107421+0000", 2024-04-19T02:03:41.428 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013848035999999999 2024-04-19T02:03:41.428 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.428 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.428 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:41.428 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.107506+0000", 2024-04-19T02:03:41.428 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5341999999999997e-05 2024-04-19T02:03:41.428 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.428 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:41.428 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.428 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.428 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.428 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:460 3.es0 3:7af77758:::smithi05930959-34:head [read 0~1867776,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:03:41.428 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.137733+0000", 2024-04-19T02:03:41.428 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 10.207005033, 2024-04-19T02:03:41.428 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013538458, 2024-04-19T02:03:41.429 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:41.429 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:41.429 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:03:41.429 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:41.429 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:41.429 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:41.429 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 460 2024-04-19T02:03:41.429 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.429 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:41.429 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.429 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:41.429 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.137733+0000", 2024-04-19T02:03:41.429 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.429 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.430 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.430 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:41.430 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.137733+0000", 2024-04-19T02:03:41.430 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:41.430 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.430 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.430 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:41.430 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.137734+0000", 2024-04-19T02:03:41.430 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3200000000000001e-06 2024-04-19T02:03:41.430 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.430 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.430 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:41.430 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.137742+0000", 2024-04-19T02:03:41.430 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.763e-06 2024-04-19T02:03:41.431 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.431 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.431 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:41.431 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.137761+0000", 2024-04-19T02:03:41.431 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9358999999999999e-05 2024-04-19T02:03:41.431 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.431 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.431 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:41.431 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.137774+0000", 2024-04-19T02:03:41.431 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3089e-05 2024-04-19T02:03:41.431 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.432 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.432 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:41.432 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.140110+0000", 2024-04-19T02:03:41.432 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0023359129999999998 2024-04-19T02:03:41.432 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.432 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.432 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:41.432 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.140354+0000", 2024-04-19T02:03:41.432 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00024421600000000001 2024-04-19T02:03:41.432 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:41.432 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:41.432 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:41.432 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.151271+0000", 2024-04-19T02:03:41.432 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010916798 2024-04-19T02:03:41.432 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.433 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:41.433 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.433 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:41.433 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:41.433 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:41.436 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-19T02:03:41.437 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:41.430+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:41.437 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-04-19T02:03:41.437 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T02:03:41.451 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-04-19T02:03:41.451 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T02:03:41.468 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-04-19T02:03:41.468 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T02:03:41.485 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-04-19T02:03:41.485 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T02:03:41.506 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-04-19T02:03:41.506 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T02:03:41.526 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-04-19T02:03:41.527 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T02:03:41.535 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:41.534+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:41.547 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-04-19T02:03:41.547 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T02:03:41.567 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:41.567 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:03:41.567 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-04-19T02:03:41.567 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:41.570 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-04-19T02:03:41.570 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T02:03:41.581 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-19T02:03:41.591 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-04-19T02:03:41.591 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T02:03:41.610 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-04-19T02:03:41.610 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T02:03:41.630 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-04-19T02:03:41.630 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T02:03:41.636 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:41.638+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:41.650 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-04-19T02:03:41.651 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T02:03:41.701 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:41.701 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:41.714 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-19T02:03:41.737 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:41.738+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:41.831 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:41.832 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:41.838 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:41.838+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:41.846 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-19T02:03:41.940 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:41.937+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:42.019 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:42.020 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:42.033 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-19T02:03:42.040 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:42.042+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:42.041 INFO:teuthology.orchestra.run.smithi059.stderr:nodeep-scrub is unset 2024-04-19T02:03:42.046 INFO:tasks.rados.rados.0.smithi059.stdout:228: done (3 left) 2024-04-19T02:03:42.046 INFO:tasks.rados.rados.0.smithi059.stdout:231: done (2 left) 2024-04-19T02:03:42.046 INFO:tasks.rados.rados.0.smithi059.stdout:232: done (1 left) 2024-04-19T02:03:42.046 INFO:tasks.rados.rados.0.smithi059.stdout:235: done (0 left) 2024-04-19T02:03:42.046 INFO:tasks.rados.rados.0.smithi059.stdout:236: snap_create 2024-04-19T02:03:42.140 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:42.138+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:42.150 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:42.150 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:42.164 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-19T02:03:42.242 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:42.242+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:42.331 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:42.331 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:42.342 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:42.342+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:42.344 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-19T02:03:42.444 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:42.446+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:42.468 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:42.468 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:42.482 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-19T02:03:42.544 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:42.545+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:42.603 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:42.603 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:42.618 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-19T02:03:42.645 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:42.641+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:42.743 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:42.743 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:42.745 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:42.746+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:42.757 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-19T02:03:42.846 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:42.845+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:42.922 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:42.922 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:42.937 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-19T02:03:42.947 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:42.949+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:43.048 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:43.045+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:43.049 INFO:tasks.rados.rados.0.smithi059.stdout:236: done (0 left) 2024-04-19T02:03:43.049 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 49 current snap is 17 2024-04-19T02:03:43.049 INFO:tasks.rados.rados.0.smithi059.stdout:237: seq_num 112 ranges {1933312=720896,2654208=802816,3457024=360448} 2024-04-19T02:03:43.086 INFO:tasks.rados.rados.0.smithi059.stdout:237: writing smithi05930959-49 from 1933312 to 2654208 tid 1 2024-04-19T02:03:43.098 INFO:tasks.rados.rados.0.smithi059.stdout:237: writing smithi05930959-49 from 2654208 to 3457024 tid 2 2024-04-19T02:03:43.102 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:43.102 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:43.104 INFO:tasks.rados.rados.0.smithi059.stdout:237: writing smithi05930959-49 from 3457024 to 3817472 tid 3 2024-04-19T02:03:43.104 INFO:tasks.rados.rados.0.smithi059.stdout:238: rmattr oid 20 current snap is 17 2024-04-19T02:03:43.104 INFO:tasks.rados.rados.0.smithi059.stdout:237: finishing write tid 1 to smithi05930959-49 2024-04-19T02:03:43.104 INFO:tasks.rados.rados.0.smithi059.stdout:239: read oid 6 snap -1 2024-04-19T02:03:43.104 INFO:tasks.rados.rados.0.smithi059.stdout:239: expect (ObjNum 5 snap 0 seq_num 5) 2024-04-19T02:03:43.104 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 48 current snap is 17 2024-04-19T02:03:43.104 INFO:tasks.rados.rados.0.smithi059.stdout:240: seq_num 113 ranges {802816=737280,1540096=557056,2097152=606208} 2024-04-19T02:03:43.123 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-19T02:03:43.124 INFO:tasks.rados.rados.0.smithi059.stdout:240: writing smithi05930959-48 from 802816 to 1540096 tid 1 2024-04-19T02:03:43.130 INFO:tasks.rados.rados.0.smithi059.stdout:240: writing smithi05930959-48 from 1540096 to 2097152 tid 2 2024-04-19T02:03:43.139 INFO:tasks.rados.rados.0.smithi059.stdout:240: writing smithi05930959-48 from 2097152 to 2703360 tid 3 2024-04-19T02:03:43.139 INFO:tasks.rados.rados.0.smithi059.stdout:241: read oid 11 snap -1 2024-04-19T02:03:43.139 INFO:tasks.rados.rados.0.smithi059.stdout:241: expect deleted 2024-04-19T02:03:43.139 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 26 current snap is 17 2024-04-19T02:03:43.139 INFO:tasks.rados.rados.0.smithi059.stdout:237: finishing write tid 2 to smithi05930959-49 2024-04-19T02:03:43.139 INFO:tasks.rados.rados.0.smithi059.stdout:242: seq_num 114 ranges {425984=753664,1179648=688128,1867776=475136} 2024-04-19T02:03:43.149 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:43.145+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:43.152 INFO:tasks.rados.rados.0.smithi059.stdout:242: writing smithi05930959-26 from 425984 to 1179648 tid 1 2024-04-19T02:03:43.160 INFO:tasks.rados.rados.0.smithi059.stdout:242: writing smithi05930959-26 from 1179648 to 1867776 tid 2 2024-04-19T02:03:43.165 INFO:tasks.rados.rados.0.smithi059.stdout:242: writing smithi05930959-26 from 1867776 to 2342912 tid 3 2024-04-19T02:03:43.165 INFO:tasks.rados.rados.0.smithi059.stdout:243: delete oid 45 current snap is 17 2024-04-19T02:03:43.165 INFO:tasks.rados.rados.0.smithi059.stdout:237: finishing write tid 3 to smithi05930959-49 2024-04-19T02:03:43.165 INFO:tasks.rados.rados.0.smithi059.stdout:239: expect (ObjNum 5 snap 0 seq_num 5) 2024-04-19T02:03:43.177 INFO:tasks.rados.rados.0.smithi059.stdout:237: finishing write tid 4 to smithi05930959-49 2024-04-19T02:03:43.177 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 20 v 64 (ObjNum 94 snap 9 seq_num 94) dirty exists 2024-04-19T02:03:43.177 INFO:tasks.rados.rados.0.smithi059.stdout:237: finishing write tid 5 to smithi05930959-49 2024-04-19T02:03:43.177 INFO:tasks.rados.rados.0.smithi059.stdout:237: oid 49 updating version 0 to 40 2024-04-19T02:03:43.177 INFO:tasks.rados.rados.0.smithi059.stdout:237: oid 49 updating version 40 to 41 2024-04-19T02:03:43.177 INFO:tasks.rados.rados.0.smithi059.stdout:237: oid 49 version 41 is already newer than 38 2024-04-19T02:03:43.177 INFO:tasks.rados.rados.0.smithi059.stdout:237: oid 49 version 41 is already newer than 39 2024-04-19T02:03:43.177 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 49 v 41 (ObjNum 112 snap 17 seq_num 112) dirty exists 2024-04-19T02:03:43.177 INFO:tasks.rados.rados.0.smithi059.stdout:237: left oid 49 (ObjNum 112 snap 17 seq_num 112) 2024-04-19T02:03:43.177 INFO:tasks.rados.rados.0.smithi059.stdout:240: finishing write tid 1 to smithi05930959-48 2024-04-19T02:03:43.177 INFO:tasks.rados.rados.0.smithi059.stdout:240: finishing write tid 2 to smithi05930959-48 2024-04-19T02:03:43.177 INFO:tasks.rados.rados.0.smithi059.stdout:240: finishing write tid 3 to smithi05930959-48 2024-04-19T02:03:43.177 INFO:tasks.rados.rados.0.smithi059.stdout:240: finishing write tid 4 to smithi05930959-48 2024-04-19T02:03:43.177 INFO:tasks.rados.rados.0.smithi059.stdout:240: finishing write tid 5 to smithi05930959-48 2024-04-19T02:03:43.178 INFO:tasks.rados.rados.0.smithi059.stdout:240: oid 48 updating version 0 to 69 2024-04-19T02:03:43.178 INFO:tasks.rados.rados.0.smithi059.stdout:240: oid 48 version 69 is already newer than 68 2024-04-19T02:03:43.178 INFO:tasks.rados.rados.0.smithi059.stdout:240: oid 48 version 69 is already newer than 66 2024-04-19T02:03:43.178 INFO:tasks.rados.rados.0.smithi059.stdout:240: oid 48 version 69 is already newer than 67 2024-04-19T02:03:43.178 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 48 v 69 (ObjNum 113 snap 17 seq_num 113) dirty exists 2024-04-19T02:03:43.178 INFO:tasks.rados.rados.0.smithi059.stdout:240: left oid 48 (ObjNum 113 snap 17 seq_num 113) 2024-04-19T02:03:43.178 INFO:tasks.rados.rados.0.smithi059.stdout:242: finishing write tid 1 to smithi05930959-26 2024-04-19T02:03:43.178 INFO:tasks.rados.rados.0.smithi059.stdout:242: finishing write tid 2 to smithi05930959-26 2024-04-19T02:03:43.178 INFO:tasks.rados.rados.0.smithi059.stdout:242: finishing write tid 3 to smithi05930959-26 2024-04-19T02:03:43.178 INFO:tasks.rados.rados.0.smithi059.stdout:242: finishing write tid 4 to smithi05930959-26 2024-04-19T02:03:43.179 INFO:tasks.rados.rados.0.smithi059.stdout:237: done (6 left) 2024-04-19T02:03:43.179 INFO:tasks.rados.rados.0.smithi059.stdout:238: done (5 left) 2024-04-19T02:03:43.179 INFO:tasks.rados.rados.0.smithi059.stdout:239: done (4 left) 2024-04-19T02:03:43.179 INFO:tasks.rados.rados.0.smithi059.stdout:240: done (3 left) 2024-04-19T02:03:43.179 INFO:tasks.rados.rados.0.smithi059.stdout:241: done (2 left) 2024-04-19T02:03:43.179 INFO:tasks.rados.rados.0.smithi059.stdout:243: done (1 left) 2024-04-19T02:03:43.179 INFO:tasks.rados.rados.0.smithi059.stdout:244: read oid 43 snap -1 2024-04-19T02:03:43.179 INFO:tasks.rados.rados.0.smithi059.stdout:244: expect (ObjNum 74 snap 6 seq_num 74) 2024-04-19T02:03:43.179 INFO:tasks.rados.rados.0.smithi059.stdout:245: delete oid 19 current snap is 17 2024-04-19T02:03:43.185 INFO:tasks.rados.rados.0.smithi059.stdout:242: finishing write tid 5 to smithi05930959-26 2024-04-19T02:03:43.185 INFO:tasks.rados.rados.0.smithi059.stdout:242: oid 26 updating version 0 to 61 2024-04-19T02:03:43.185 INFO:tasks.rados.rados.0.smithi059.stdout:242: oid 26 version 61 is already newer than 59 2024-04-19T02:03:43.185 INFO:tasks.rados.rados.0.smithi059.stdout:242: oid 26 version 61 is already newer than 58 2024-04-19T02:03:43.185 INFO:tasks.rados.rados.0.smithi059.stdout:242: oid 26 version 61 is already newer than 60 2024-04-19T02:03:43.185 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 26 v 61 (ObjNum 114 snap 17 seq_num 114) dirty exists 2024-04-19T02:03:43.185 INFO:tasks.rados.rados.0.smithi059.stdout:242: left oid 26 (ObjNum 114 snap 17 seq_num 114) 2024-04-19T02:03:43.185 INFO:tasks.rados.rados.0.smithi059.stdout:242: done (2 left) 2024-04-19T02:03:43.185 INFO:tasks.rados.rados.0.smithi059.stdout:245: done (1 left) 2024-04-19T02:03:43.185 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 42 current snap is 17 2024-04-19T02:03:43.185 INFO:tasks.rados.rados.0.smithi059.stdout:246: seq_num 117 ranges {1163264=425984,1589248=606208,2195456=540672,2736128=409600} 2024-04-19T02:03:43.201 INFO:tasks.rados.rados.0.smithi059.stdout:246: writing smithi05930959-42 from 1163264 to 1589248 tid 1 2024-04-19T02:03:43.207 INFO:tasks.rados.rados.0.smithi059.stdout:246: writing smithi05930959-42 from 1589248 to 2195456 tid 2 2024-04-19T02:03:43.212 INFO:tasks.rados.rados.0.smithi059.stdout:246: writing smithi05930959-42 from 2195456 to 2736128 tid 3 2024-04-19T02:03:43.216 INFO:tasks.rados.rados.0.smithi059.stdout:246: writing smithi05930959-42 from 2736128 to 3145728 tid 4 2024-04-19T02:03:43.216 INFO:tasks.rados.rados.0.smithi059.stdout:247: snap_remove snap 15 2024-04-19T02:03:43.250 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:43.249+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:43.293 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:43.293 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:43.308 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-19T02:03:43.355 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:43.358+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:43.442 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:43.442 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:43.452 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:43.453+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:43.456 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-19T02:03:43.553 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:43.553+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:43.749 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T02:03:43.750 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:43.750 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:43.751 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-19T02:03:43.751 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:43.750+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:43.851 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:43.849+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:43.936 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:43.936 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:43.952 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:43.949+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:43.953 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-19T02:03:43.963 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-04-19T02:03:43.964 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T02:03:43.980 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-04-19T02:03:43.981 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T02:03:43.997 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-04-19T02:03:43.997 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T02:03:44.014 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-04-19T02:03:44.015 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T02:03:44.035 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-04-19T02:03:44.035 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T02:03:44.053 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:44.054+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:44.056 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-04-19T02:03:44.056 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T02:03:44.069 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:44.069 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:03:44.069 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-04-19T02:03:44.069 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:44.072 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-19T02:03:44.078 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-04-19T02:03:44.078 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T02:03:44.084 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-19T02:03:44.099 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-04-19T02:03:44.100 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T02:03:44.121 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-04-19T02:03:44.121 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T02:03:44.142 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-04-19T02:03:44.143 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T02:03:44.154 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:44.153+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:44.163 INFO:tasks.rados.rados.0.smithi059.stdout:247: done (2 left) 2024-04-19T02:03:44.163 INFO:tasks.rados.rados.0.smithi059.stdout:248: snap_create 2024-04-19T02:03:44.163 INFO:tasks.rados.rados.0.smithi059.stdout:244: expect (ObjNum 74 snap 6 seq_num 74) 2024-04-19T02:03:44.181 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-04-19T02:03:44.181 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T02:03:44.217 INFO:tasks.rados.rados.0.smithi059.stdout:246: finishing write tid 1 to smithi05930959-42 2024-04-19T02:03:44.217 INFO:tasks.rados.rados.0.smithi059.stdout:246: finishing write tid 2 to smithi05930959-42 2024-04-19T02:03:44.217 INFO:tasks.rados.rados.0.smithi059.stdout:246: finishing write tid 3 to smithi05930959-42 2024-04-19T02:03:44.217 INFO:tasks.rados.rados.0.smithi059.stdout:246: finishing write tid 4 to smithi05930959-42 2024-04-19T02:03:44.217 INFO:tasks.rados.rados.0.smithi059.stdout:246: finishing write tid 5 to smithi05930959-42 2024-04-19T02:03:44.217 INFO:tasks.rados.rados.0.smithi059.stdout:246: finishing write tid 6 to smithi05930959-42 2024-04-19T02:03:44.217 INFO:tasks.rados.rados.0.smithi059.stdout:246: oid 42 updating version 0 to 47 2024-04-19T02:03:44.217 INFO:tasks.rados.rados.0.smithi059.stdout:246: oid 42 version 47 is already newer than 46 2024-04-19T02:03:44.217 INFO:tasks.rados.rados.0.smithi059.stdout:246: oid 42 updating version 47 to 48 2024-04-19T02:03:44.217 INFO:tasks.rados.rados.0.smithi059.stdout:246: oid 42 version 48 is already newer than 45 2024-04-19T02:03:44.217 INFO:tasks.rados.rados.0.smithi059.stdout:246: oid 42 updating version 48 to 49 2024-04-19T02:03:44.217 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 42 v 49 (ObjNum 117 snap 17 seq_num 117) dirty exists 2024-04-19T02:03:44.218 INFO:tasks.rados.rados.0.smithi059.stdout:246: left oid 42 (ObjNum 117 snap 17 seq_num 117) 2024-04-19T02:03:44.225 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-04-19T02:03:44.225 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T02:03:44.249 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:44.250 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:03:44.250 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-04-19T02:03:44.250 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-04-19T02:03:44.250 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:44.255 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:44.258+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:44.265 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-19T02:03:44.359 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:44.362+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:44.429 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:44.429 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-04-19T02:03:44.429 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-04-19T02:03:44.430 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-04-19T02:03:44.430 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.430 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:344 3.4s0 3:2277bf56:::smithi05930959-44:head [append 0~606208] snapc 8=[8,6,5] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:44.430 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.735396+0000", 2024-04-19T02:03:44.430 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 19.693610481, 2024-04-19T02:03:44.430 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.014124197999999999, 2024-04-19T02:03:44.430 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:44.430 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:44.430 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:44.430 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:44.430 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:44.430 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:44.431 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 344 2024-04-19T02:03:44.431 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.431 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:44.431 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.431 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:44.431 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.735396+0000", 2024-04-19T02:03:44.431 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.431 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.431 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.431 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:44.431 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.735396+0000", 2024-04-19T02:03:44.431 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.431 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.432 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.432 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:44.432 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.735399+0000", 2024-04-19T02:03:44.432 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.32e-06 2024-04-19T02:03:44.432 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.432 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.432 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:44.432 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.736033+0000", 2024-04-19T02:03:44.432 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000633689 2024-04-19T02:03:44.432 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.432 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.432 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:44.433 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.736074+0000", 2024-04-19T02:03:44.433 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1020999999999999e-05 2024-04-19T02:03:44.433 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.433 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.433 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:44.433 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.736094+0000", 2024-04-19T02:03:44.433 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0339e-05 2024-04-19T02:03:44.433 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.433 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.433 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:44.433 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.736202+0000", 2024-04-19T02:03:44.433 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000107899 2024-04-19T02:03:44.433 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.433 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.434 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "op must wait for map", 2024-04-19T02:03:44.434 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.736212+0000", 2024-04-19T02:03:44.434 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.7259999999999997e-06 2024-04-19T02:03:44.434 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.434 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.434 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:44.434 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.738290+0000", 2024-04-19T02:03:44.434 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002078725 2024-04-19T02:03:44.434 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.434 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.434 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:44.434 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.738359+0000", 2024-04-19T02:03:44.435 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8177999999999997e-05 2024-04-19T02:03:44.435 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.435 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.435 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:44.435 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.739412+0000", 2024-04-19T02:03:44.435 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0010534240000000001 2024-04-19T02:03:44.435 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.435 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.435 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:44.435 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.741318+0000", 2024-04-19T02:03:44.435 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001906151 2024-04-19T02:03:44.435 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.435 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.436 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:44.436 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749459+0000", 2024-04-19T02:03:44.436 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0081410830000000003 2024-04-19T02:03:44.436 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.436 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.436 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:44.436 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749520+0000", 2024-04-19T02:03:44.436 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0643000000000002e-05 2024-04-19T02:03:44.436 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.436 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:44.436 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.436 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.437 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.437 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:345 3.4s0 3:2277bf56:::smithi05930959-44:head [append 0~720896] snapc 8=[8,6,5] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:44.437 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.745708+0000", 2024-04-19T02:03:44.437 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 19.683297927000002, 2024-04-19T02:03:44.437 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0064257389999999998, 2024-04-19T02:03:44.437 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:44.437 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:44.437 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:44.437 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:44.437 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:44.437 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:44.437 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 345 2024-04-19T02:03:44.437 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.438 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:44.438 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.438 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:44.438 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.745708+0000", 2024-04-19T02:03:44.438 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.438 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.438 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.438 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:44.438 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.745708+0000", 2024-04-19T02:03:44.438 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.438 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.438 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.439 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:44.439 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.745710+0000", 2024-04-19T02:03:44.439 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6619999999999999e-06 2024-04-19T02:03:44.439 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.439 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.439 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:44.439 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746146+0000", 2024-04-19T02:03:44.439 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00043585500000000002 2024-04-19T02:03:44.439 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.439 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.439 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:44.439 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746160+0000", 2024-04-19T02:03:44.439 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4460999999999999e-05 2024-04-19T02:03:44.439 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.440 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.440 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:44.440 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746169+0000", 2024-04-19T02:03:44.440 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.9369999999999993e-06 2024-04-19T02:03:44.440 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.440 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.440 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:44.440 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746255+0000", 2024-04-19T02:03:44.440 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5756999999999999e-05 2024-04-19T02:03:44.440 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.440 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.440 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:44.440 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746358+0000", 2024-04-19T02:03:44.441 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000102722 2024-04-19T02:03:44.441 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.441 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.441 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:44.441 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.747578+0000", 2024-04-19T02:03:44.441 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001220643 2024-04-19T02:03:44.441 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.441 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.441 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:44.441 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749531+0000", 2024-04-19T02:03:44.441 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001952454 2024-04-19T02:03:44.441 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.441 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.442 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:44.442 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.752059+0000", 2024-04-19T02:03:44.442 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002528218 2024-04-19T02:03:44.442 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.442 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.442 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:44.442 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.752134+0000", 2024-04-19T02:03:44.442 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.5030000000000005e-05 2024-04-19T02:03:44.442 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.442 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:44.442 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.442 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.442 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.443 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:346 3.4s0 3:2277bf56:::smithi05930959-44:head [append 0~49152] snapc 8=[8,6,5] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:44.443 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.746194+0000", 2024-04-19T02:03:44.443 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 19.682812246000001, 2024-04-19T02:03:44.443 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.009380457, 2024-04-19T02:03:44.443 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:44.443 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:44.443 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:44.443 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:44.443 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:44.443 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:44.443 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 346 2024-04-19T02:03:44.444 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.444 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:44.444 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.444 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:44.444 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746194+0000", 2024-04-19T02:03:44.444 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.444 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.444 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.444 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:44.444 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746194+0000", 2024-04-19T02:03:44.444 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.444 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.445 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.445 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:44.445 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746195+0000", 2024-04-19T02:03:44.445 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.6199999999999997e-07 2024-04-19T02:03:44.445 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.445 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.445 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:44.445 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746238+0000", 2024-04-19T02:03:44.445 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.3105999999999997e-05 2024-04-19T02:03:44.445 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.445 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.445 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:44.446 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746247+0000", 2024-04-19T02:03:44.446 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.1540000000000008e-06 2024-04-19T02:03:44.446 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.446 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.446 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:44.446 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746257+0000", 2024-04-19T02:03:44.446 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0558e-05 2024-04-19T02:03:44.446 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.446 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.446 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:44.446 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.748250+0000", 2024-04-19T02:03:44.446 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0019923380000000002 2024-04-19T02:03:44.447 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.447 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.447 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:44.447 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.748349+0000", 2024-04-19T02:03:44.447 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9030999999999994e-05 2024-04-19T02:03:44.447 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.447 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.447 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:44.447 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.748987+0000", 2024-04-19T02:03:44.447 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00063871800000000003 2024-04-19T02:03:44.447 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.447 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.447 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:44.448 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750226+0000", 2024-04-19T02:03:44.448 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0012389250000000001 2024-04-19T02:03:44.448 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.448 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.448 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:44.448 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.755523+0000", 2024-04-19T02:03:44.448 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.005297024 2024-04-19T02:03:44.448 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.448 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.448 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:44.448 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.755574+0000", 2024-04-19T02:03:44.448 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.0940999999999999e-05 2024-04-19T02:03:44.448 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.449 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:44.449 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.449 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.449 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.449 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:347 3.4s0 3:2277bf56:::smithi05930959-44:head [setxattr _header (56) in=7b] snapc 8=[8,6,5] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:44.449 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.746274+0000", 2024-04-19T02:03:44.449 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 19.682732066, 2024-04-19T02:03:44.449 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010236558, 2024-04-19T02:03:44.449 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:44.449 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:44.449 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:44.449 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:44.449 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:44.450 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:44.450 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 347 2024-04-19T02:03:44.450 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.450 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:44.450 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.450 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:44.450 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746274+0000", 2024-04-19T02:03:44.450 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.450 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.450 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.450 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:44.450 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746274+0000", 2024-04-19T02:03:44.451 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.451 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.451 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.451 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:44.451 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746275+0000", 2024-04-19T02:03:44.451 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.3900000000000004e-07 2024-04-19T02:03:44.451 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.451 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.451 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:44.451 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746279+0000", 2024-04-19T02:03:44.451 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0679999999999998e-06 2024-04-19T02:03:44.451 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.451 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.451 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:44.452 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746299+0000", 2024-04-19T02:03:44.452 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9641e-05 2024-04-19T02:03:44.452 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.452 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.452 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:44.452 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746307+0000", 2024-04-19T02:03:44.452 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2570000000000007e-06 2024-04-19T02:03:44.452 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.452 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.452 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:44.452 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749555+0000", 2024-04-19T02:03:44.452 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0032478519999999999 2024-04-19T02:03:44.453 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.453 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.453 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:44.453 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749610+0000", 2024-04-19T02:03:44.453 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5155999999999998e-05 2024-04-19T02:03:44.453 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.453 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.453 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:44.453 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749969+0000", 2024-04-19T02:03:44.453 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00035909400000000001 2024-04-19T02:03:44.453 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.453 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.453 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:44.454 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750800+0000", 2024-04-19T02:03:44.454 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00083078000000000004 2024-04-19T02:03:44.454 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.454 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.454 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:44.454 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.756433+0000", 2024-04-19T02:03:44.454 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0056334009999999997 2024-04-19T02:03:44.454 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.454 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.454 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:44.454 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.756511+0000", 2024-04-19T02:03:44.454 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.7570000000000004e-05 2024-04-19T02:03:44.454 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.455 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:44.455 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.455 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.455 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.455 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:348 3.4s0 3:2277bf56:::smithi05930959-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:44.455 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.746345+0000", 2024-04-19T02:03:44.455 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 19.682661261, 2024-04-19T02:03:44.455 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012391231000000001, 2024-04-19T02:03:44.455 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:44.455 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:44.455 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:44.455 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:44.455 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:44.456 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:44.456 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 348 2024-04-19T02:03:44.456 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.456 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:44.456 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.456 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:44.456 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746345+0000", 2024-04-19T02:03:44.456 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.456 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.456 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.456 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:44.456 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746345+0000", 2024-04-19T02:03:44.456 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.457 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.457 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.457 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:44.457 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746346+0000", 2024-04-19T02:03:44.457 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.5899999999999996e-07 2024-04-19T02:03:44.457 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.457 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.457 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:44.457 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746349+0000", 2024-04-19T02:03:44.457 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2100000000000002e-06 2024-04-19T02:03:44.457 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.457 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.457 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:44.458 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746359+0000", 2024-04-19T02:03:44.458 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.8069999999999994e-06 2024-04-19T02:03:44.458 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.458 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.458 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:44.458 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746366+0000", 2024-04-19T02:03:44.458 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.1969999999999999e-06 2024-04-19T02:03:44.458 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.458 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.458 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:44.458 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750173+0000", 2024-04-19T02:03:44.458 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0038071400000000001 2024-04-19T02:03:44.459 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.459 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.459 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:44.459 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750223+0000", 2024-04-19T02:03:44.459 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9597e-05 2024-04-19T02:03:44.459 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.459 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.459 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:44.459 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.757010+0000", 2024-04-19T02:03:44.459 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.006787062 2024-04-19T02:03:44.459 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.459 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.459 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:44.459 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.757155+0000", 2024-04-19T02:03:44.460 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000145303 2024-04-19T02:03:44.460 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.460 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.460 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:44.460 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.758736+0000", 2024-04-19T02:03:44.460 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0015812560000000001 2024-04-19T02:03:44.460 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.460 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:44.460 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.460 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.460 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.460 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:374 3.4s0 3:2277bf56:::smithi05930959-44:head [append 0~0] snapc 9=[9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:44.461 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:26.758473+0000", 2024-04-19T02:03:44.461 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 17.670533014, 2024-04-19T02:03:44.461 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.032817486999999999, 2024-04-19T02:03:44.461 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:44.461 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:44.461 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:44.461 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:44.461 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:44.461 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:44.461 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 374 2024-04-19T02:03:44.461 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.461 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:44.461 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.461 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:44.462 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758473+0000", 2024-04-19T02:03:44.462 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.462 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.462 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.462 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:44.462 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758473+0000", 2024-04-19T02:03:44.462 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.462 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.462 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.462 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:44.462 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758475+0000", 2024-04-19T02:03:44.462 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.835e-06 2024-04-19T02:03:44.463 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.463 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.463 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:44.463 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758481+0000", 2024-04-19T02:03:44.463 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0669999999999997e-06 2024-04-19T02:03:44.463 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.463 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.463 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:44.463 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758497+0000", 2024-04-19T02:03:44.463 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6195e-05 2024-04-19T02:03:44.463 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.463 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.464 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:44.464 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758506+0000", 2024-04-19T02:03:44.464 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.1730000000000001e-06 2024-04-19T02:03:44.464 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.464 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.464 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:44.464 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758624+0000", 2024-04-19T02:03:44.464 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000117336 2024-04-19T02:03:44.464 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.464 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.464 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:44.464 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758812+0000", 2024-04-19T02:03:44.465 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000188684 2024-04-19T02:03:44.465 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.465 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.465 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:44.465 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.760058+0000", 2024-04-19T02:03:44.465 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001245812 2024-04-19T02:03:44.465 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.465 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.465 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:44.465 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.762381+0000", 2024-04-19T02:03:44.465 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0023226549999999999 2024-04-19T02:03:44.465 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.465 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.466 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:44.466 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.791180+0000", 2024-04-19T02:03:44.466 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.028798727 2024-04-19T02:03:44.466 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.466 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.466 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:44.466 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.791291+0000", 2024-04-19T02:03:44.466 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00011100299999999999 2024-04-19T02:03:44.466 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.466 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:44.466 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.466 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.467 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.467 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:375 3.4s0 3:2277bf56:::smithi05930959-44:head [rollback 8] snapc 9=[9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:44.467 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:26.758531+0000", 2024-04-19T02:03:44.467 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 17.670475141000001, 2024-04-19T02:03:44.467 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.033180481999999997, 2024-04-19T02:03:44.467 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:44.467 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:44.467 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:44.467 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:44.467 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:44.467 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:44.467 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 375 2024-04-19T02:03:44.467 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.467 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:44.468 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.468 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:44.468 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758531+0000", 2024-04-19T02:03:44.468 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.468 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.468 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.468 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:44.468 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758531+0000", 2024-04-19T02:03:44.468 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.468 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.468 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.468 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:44.468 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758532+0000", 2024-04-19T02:03:44.469 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.5700000000000002e-07 2024-04-19T02:03:44.469 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.469 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.469 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:44.469 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758535+0000", 2024-04-19T02:03:44.469 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4520000000000002e-06 2024-04-19T02:03:44.469 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.469 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.469 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:44.469 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758546+0000", 2024-04-19T02:03:44.469 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1195e-05 2024-04-19T02:03:44.469 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.469 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.470 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:44.470 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758554+0000", 2024-04-19T02:03:44.470 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.2579999999999998e-06 2024-04-19T02:03:44.470 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.470 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.470 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:44.470 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.760997+0000", 2024-04-19T02:03:44.470 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0024429930000000001 2024-04-19T02:03:44.470 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.470 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.470 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:44.470 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.761153+0000", 2024-04-19T02:03:44.470 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00015617100000000001 2024-04-19T02:03:44.471 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.471 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.471 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:44.471 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.761823+0000", 2024-04-19T02:03:44.471 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00067058000000000005 2024-04-19T02:03:44.471 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.471 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.471 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:44.471 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.765008+0000", 2024-04-19T02:03:44.471 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0031845300000000001 2024-04-19T02:03:44.471 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.471 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.471 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:44.472 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.791630+0000", 2024-04-19T02:03:44.472 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.026622456999999999 2024-04-19T02:03:44.472 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.472 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.472 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:44.472 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.791712+0000", 2024-04-19T02:03:44.472 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.1188999999999998e-05 2024-04-19T02:03:44.472 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.472 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:44.472 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.472 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.472 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.472 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:376 3.4s0 3:2277bf56:::smithi05930959-44:head [append 0~0] snapc 9=[9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:44.473 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:26.758565+0000", 2024-04-19T02:03:44.473 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 17.670441383, 2024-04-19T02:03:44.473 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.033486091000000003, 2024-04-19T02:03:44.473 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:44.473 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:44.473 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:44.473 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:44.473 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:44.473 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:44.473 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 376 2024-04-19T02:03:44.473 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.473 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:44.473 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.473 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:44.474 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758565+0000", 2024-04-19T02:03:44.474 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.474 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.474 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.474 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:44.474 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758565+0000", 2024-04-19T02:03:44.474 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.474 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.474 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.474 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:44.474 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758565+0000", 2024-04-19T02:03:44.474 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9900000000000001e-07 2024-04-19T02:03:44.475 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.475 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.475 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:44.475 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758568+0000", 2024-04-19T02:03:44.475 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0010000000000002e-06 2024-04-19T02:03:44.475 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.475 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.475 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:44.475 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758576+0000", 2024-04-19T02:03:44.475 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.9719999999999992e-06 2024-04-19T02:03:44.475 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.475 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.475 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:44.476 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758583+0000", 2024-04-19T02:03:44.476 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.2489999999999998e-06 2024-04-19T02:03:44.476 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.476 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.476 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:44.476 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.763053+0000", 2024-04-19T02:03:44.476 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0044703809999999998 2024-04-19T02:03:44.476 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.476 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.476 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:44.476 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.763266+0000", 2024-04-19T02:03:44.476 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00021301299999999999 2024-04-19T02:03:44.476 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.477 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.477 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:44.477 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.764685+0000", 2024-04-19T02:03:44.477 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0014195379999999999 2024-04-19T02:03:44.477 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.477 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.477 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:44.477 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.765894+0000", 2024-04-19T02:03:44.477 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0012088859999999999 2024-04-19T02:03:44.478 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.478 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.478 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:44.478 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.791938+0000", 2024-04-19T02:03:44.478 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.026043251999999999 2024-04-19T02:03:44.478 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.478 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.478 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:44.478 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.792051+0000", 2024-04-19T02:03:44.478 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0001133 2024-04-19T02:03:44.478 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.478 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:44.478 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.479 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.479 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.479 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.as2 46/26 ECSubWrite(tid=102, reqid=osd.11.0:102, at_version=46'47, trim_to=0'0, roll_forward_to=45'45))", 2024-04-19T02:03:44.479 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.061978+0000", 2024-04-19T02:03:44.479 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 14.367028256999999, 2024-04-19T02:03:44.479 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0086516360000000007, 2024-04-19T02:03:44.479 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:44.479 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:44.479 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:44.479 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:44.479 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.479 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:44.479 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.061978+0000", 2024-04-19T02:03:44.480 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.480 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.480 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.480 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:44.480 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.061978+0000", 2024-04-19T02:03:44.480 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.480 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.480 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.480 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:44.480 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.061979+0000", 2024-04-19T02:03:44.480 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.8000000000000004e-07 2024-04-19T02:03:44.481 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.481 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.481 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:44.481 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.062004+0000", 2024-04-19T02:03:44.481 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.531e-05 2024-04-19T02:03:44.481 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.481 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.481 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:44.481 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.062022+0000", 2024-04-19T02:03:44.481 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8000999999999999e-05 2024-04-19T02:03:44.481 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.481 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.481 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:44.482 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.062036+0000", 2024-04-19T02:03:44.482 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3499999999999999e-05 2024-04-19T02:03:44.482 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.482 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.482 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:44.482 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.062138+0000", 2024-04-19T02:03:44.482 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010238099999999999 2024-04-19T02:03:44.482 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.482 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.482 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:44.482 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.062149+0000", 2024-04-19T02:03:44.482 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0706e-05 2024-04-19T02:03:44.483 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.483 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.483 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:44.483 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.070585+0000", 2024-04-19T02:03:44.483 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0084365910000000002 2024-04-19T02:03:44.483 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.483 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.483 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:44.483 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.070630+0000", 2024-04-19T02:03:44.483 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4267000000000002e-05 2024-04-19T02:03:44.483 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.483 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:44.483 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.484 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.484 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.484 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.8s1 46/26 ECSubWrite(tid=81, reqid=osd.4.0:81, at_version=46'42, trim_to=0'0, roll_forward_to=45'40))", 2024-04-19T02:03:44.484 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.062231+0000", 2024-04-19T02:03:44.484 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 14.366775262999999, 2024-04-19T02:03:44.484 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0074011800000000003, 2024-04-19T02:03:44.484 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:44.484 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:44.484 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:44.484 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:44.484 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.484 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:44.484 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.062231+0000", 2024-04-19T02:03:44.485 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.485 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.485 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.485 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:44.485 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.062231+0000", 2024-04-19T02:03:44.485 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.485 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.485 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.485 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:44.485 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.062231+0000", 2024-04-19T02:03:44.485 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.96e-07 2024-04-19T02:03:44.485 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.485 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.485 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:44.486 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.062253+0000", 2024-04-19T02:03:44.486 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2266e-05 2024-04-19T02:03:44.486 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.486 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.486 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:44.486 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.062266+0000", 2024-04-19T02:03:44.486 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2784e-05 2024-04-19T02:03:44.486 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.486 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.486 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:44.486 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.062275+0000", 2024-04-19T02:03:44.486 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7069999999999998e-06 2024-04-19T02:03:44.487 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.487 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.487 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:44.487 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.062378+0000", 2024-04-19T02:03:44.487 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000103473 2024-04-19T02:03:44.487 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.487 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.487 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:44.487 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.062388+0000", 2024-04-19T02:03:44.487 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9170000000000001e-06 2024-04-19T02:03:44.487 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.487 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.487 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:44.487 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.069586+0000", 2024-04-19T02:03:44.488 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0071980660000000004 2024-04-19T02:03:44.488 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.488 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.488 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:44.488 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.069632+0000", 2024-04-19T02:03:44.488 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5670999999999997e-05 2024-04-19T02:03:44.488 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.488 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:44.488 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.488 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.488 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.488 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.4s0 46/26 ECSubWriteReply(tid=58, last_complete=46'44, committed=1, applied=1))", 2024-04-19T02:03:44.488 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.063287+0000", 2024-04-19T02:03:44.489 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 14.36571938, 2024-04-19T02:03:44.489 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0063228360000000001, 2024-04-19T02:03:44.489 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:44.489 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:44.489 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:44.489 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:44.489 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.489 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:44.489 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.063287+0000", 2024-04-19T02:03:44.489 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.489 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.489 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.489 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:44.490 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.063287+0000", 2024-04-19T02:03:44.490 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.490 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.490 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.490 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:44.490 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.063288+0000", 2024-04-19T02:03:44.490 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2099999999999995e-07 2024-04-19T02:03:44.490 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.490 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.490 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:44.490 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.063298+0000", 2024-04-19T02:03:44.490 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0344e-05 2024-04-19T02:03:44.490 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.491 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.491 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:44.491 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.063313+0000", 2024-04-19T02:03:44.491 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4949e-05 2024-04-19T02:03:44.491 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.491 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.491 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:44.491 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.063321+0000", 2024-04-19T02:03:44.491 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5760000000000006e-06 2024-04-19T02:03:44.491 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.491 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.491 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:44.492 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.069554+0000", 2024-04-19T02:03:44.492 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0062328599999999998 2024-04-19T02:03:44.492 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.492 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.492 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:44.492 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.069610+0000", 2024-04-19T02:03:44.492 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5285999999999999e-05 2024-04-19T02:03:44.492 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.492 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:44.492 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.492 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.492 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.492 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.as2 46/26 ECSubWrite(tid=103, reqid=osd.11.0:103, at_version=46'49, trim_to=0'0, roll_forward_to=45'45))", 2024-04-19T02:03:44.493 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.069253+0000", 2024-04-19T02:03:44.493 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 14.359753657000001, 2024-04-19T02:03:44.493 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0096013669999999995, 2024-04-19T02:03:44.493 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:44.493 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:44.493 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:44.493 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:44.493 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.493 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:44.493 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.069253+0000", 2024-04-19T02:03:44.493 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.493 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.494 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.494 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:44.494 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.069253+0000", 2024-04-19T02:03:44.494 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.494 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.494 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.494 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:44.494 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.069253+0000", 2024-04-19T02:03:44.494 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.0100000000000004e-07 2024-04-19T02:03:44.494 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.494 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.494 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:44.494 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.069282+0000", 2024-04-19T02:03:44.495 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8617999999999999e-05 2024-04-19T02:03:44.495 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.495 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.495 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:44.495 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.069293+0000", 2024-04-19T02:03:44.495 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1276e-05 2024-04-19T02:03:44.495 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.495 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.495 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:44.495 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.069302+0000", 2024-04-19T02:03:44.495 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.8910000000000001e-06 2024-04-19T02:03:44.495 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.495 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.496 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:44.496 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.069822+0000", 2024-04-19T02:03:44.496 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00051964899999999996 2024-04-19T02:03:44.496 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.496 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.496 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:44.496 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.069835+0000", 2024-04-19T02:03:44.496 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.358e-05 2024-04-19T02:03:44.496 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.496 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.496 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:44.496 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.078808+0000", 2024-04-19T02:03:44.496 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0089730929999999997 2024-04-19T02:03:44.497 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.497 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.497 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:44.497 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.078854+0000", 2024-04-19T02:03:44.497 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5559000000000002e-05 2024-04-19T02:03:44.497 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.497 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:44.497 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.497 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.497 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.497 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.7s7 46/26 ECSubWrite(tid=158, reqid=osd.3.0:158, at_version=46'14, trim_to=0'0, roll_forward_to=45'12))", 2024-04-19T02:03:44.497 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.071853+0000", 2024-04-19T02:03:44.497 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 14.357152749000001, 2024-04-19T02:03:44.498 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.009672043, 2024-04-19T02:03:44.498 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:44.498 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:44.498 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:44.498 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:44.498 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.498 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:44.498 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.071853+0000", 2024-04-19T02:03:44.498 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.498 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.498 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.498 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:44.499 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.071853+0000", 2024-04-19T02:03:44.499 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.499 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.499 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.499 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:44.499 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.071854+0000", 2024-04-19T02:03:44.499 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1600000000000002e-07 2024-04-19T02:03:44.499 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.499 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.499 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:44.499 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.071880+0000", 2024-04-19T02:03:44.499 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6648e-05 2024-04-19T02:03:44.499 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.500 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.500 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:44.500 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.071895+0000", 2024-04-19T02:03:44.500 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4491999999999999e-05 2024-04-19T02:03:44.500 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.500 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.500 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:44.500 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.071915+0000", 2024-04-19T02:03:44.500 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9762000000000002e-05 2024-04-19T02:03:44.500 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.500 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.500 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:44.500 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.071997+0000", 2024-04-19T02:03:44.501 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2693000000000003e-05 2024-04-19T02:03:44.501 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.501 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.501 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:44.501 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.072009+0000", 2024-04-19T02:03:44.501 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2004000000000001e-05 2024-04-19T02:03:44.501 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.501 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.501 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:44.501 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.081483+0000", 2024-04-19T02:03:44.501 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0094739990000000003 2024-04-19T02:03:44.501 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.501 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.502 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:44.502 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.081525+0000", 2024-04-19T02:03:44.502 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2029e-05 2024-04-19T02:03:44.502 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.502 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:44.502 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.502 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.502 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.502 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.2s5 46/26 ECSubWrite(tid=159, reqid=osd.3.0:159, at_version=46'75, trim_to=0'0, roll_forward_to=45'73))", 2024-04-19T02:03:44.502 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.076790+0000", 2024-04-19T02:03:44.502 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 14.352215693, 2024-04-19T02:03:44.502 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0064720070000000001, 2024-04-19T02:03:44.502 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:44.503 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:44.503 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:44.503 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:44.503 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.503 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:44.503 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.076790+0000", 2024-04-19T02:03:44.503 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.503 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.503 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.503 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:44.503 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.076790+0000", 2024-04-19T02:03:44.503 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.503 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.504 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.504 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:44.504 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.076791+0000", 2024-04-19T02:03:44.504 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.1999999999999998e-07 2024-04-19T02:03:44.504 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.504 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.504 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:44.504 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.076823+0000", 2024-04-19T02:03:44.504 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1272999999999997e-05 2024-04-19T02:03:44.504 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.504 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.504 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:44.505 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.076837+0000", 2024-04-19T02:03:44.505 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4329e-05 2024-04-19T02:03:44.505 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.505 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.505 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:44.505 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.076847+0000", 2024-04-19T02:03:44.505 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0125999999999999e-05 2024-04-19T02:03:44.505 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.505 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.505 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:44.505 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.076953+0000", 2024-04-19T02:03:44.505 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000106254 2024-04-19T02:03:44.505 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.506 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.506 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:44.506 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.076965+0000", 2024-04-19T02:03:44.506 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1270999999999999e-05 2024-04-19T02:03:44.506 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.506 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.506 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:44.506 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.083216+0000", 2024-04-19T02:03:44.506 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0062513739999999996 2024-04-19T02:03:44.506 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.506 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.506 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:44.506 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.083262+0000", 2024-04-19T02:03:44.507 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.6560000000000001e-05 2024-04-19T02:03:44.507 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.507 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:44.507 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.507 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.507 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.507 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.1s7 46/26 ECSubWrite(tid=27, reqid=client.4509.0:447, at_version=46'13, trim_to=0'0, roll_forward_to=46'11))", 2024-04-19T02:03:44.507 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.227925+0000", 2024-04-19T02:03:44.507 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 14.201081489, 2024-04-19T02:03:44.507 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00638954, 2024-04-19T02:03:44.507 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:44.507 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:44.507 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:44.508 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:44.508 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.508 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:44.508 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.227925+0000", 2024-04-19T02:03:44.508 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.508 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.508 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.508 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:44.508 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.227925+0000", 2024-04-19T02:03:44.508 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.508 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.508 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.508 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:44.509 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.227926+0000", 2024-04-19T02:03:44.509 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0049999999999999e-06 2024-04-19T02:03:44.509 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.509 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.509 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:44.509 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.227947+0000", 2024-04-19T02:03:44.509 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1141999999999998e-05 2024-04-19T02:03:44.509 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.509 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.509 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:44.509 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.227959+0000", 2024-04-19T02:03:44.509 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2438e-05 2024-04-19T02:03:44.509 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.510 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.510 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:44.510 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.227969+0000", 2024-04-19T02:03:44.510 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.6129999999999993e-06 2024-04-19T02:03:44.510 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.510 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.510 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:44.510 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.228043+0000", 2024-04-19T02:03:44.510 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.3800999999999994e-05 2024-04-19T02:03:44.510 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.510 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.510 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:44.511 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.228053+0000", 2024-04-19T02:03:44.511 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0141e-05 2024-04-19T02:03:44.511 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.511 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.511 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:44.511 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.234258+0000", 2024-04-19T02:03:44.511 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.006204775 2024-04-19T02:03:44.511 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.511 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.511 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:44.511 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.234314+0000", 2024-04-19T02:03:44.511 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.6625000000000001e-05 2024-04-19T02:03:44.511 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.512 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:44.512 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.512 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.512 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.512 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:481 3.4s0 3:2859204c:::smithi05930959-39:head [append 0~540672] snapc a=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:03:44.512 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:31.209807+0000", 2024-04-19T02:03:44.512 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 13.219199355000001, 2024-04-19T02:03:44.512 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.007105643, 2024-04-19T02:03:44.512 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:44.512 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:44.512 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:44.512 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:44.512 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:44.513 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:44.513 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 481 2024-04-19T02:03:44.513 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.513 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:44.513 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.513 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:44.513 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.209807+0000", 2024-04-19T02:03:44.513 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.513 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.513 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.513 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:44.513 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.209807+0000", 2024-04-19T02:03:44.513 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.514 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.514 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.514 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:44.514 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.209809+0000", 2024-04-19T02:03:44.514 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9e-06 2024-04-19T02:03:44.514 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.514 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.514 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:44.514 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.210137+0000", 2024-04-19T02:03:44.514 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00032858900000000001 2024-04-19T02:03:44.514 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.514 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.514 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:44.515 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.210150+0000", 2024-04-19T02:03:44.515 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2965e-05 2024-04-19T02:03:44.515 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.515 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.515 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:44.515 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.210156+0000", 2024-04-19T02:03:44.515 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.468e-06 2024-04-19T02:03:44.515 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.515 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.515 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:44.515 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.210199+0000", 2024-04-19T02:03:44.515 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.3022000000000002e-05 2024-04-19T02:03:44.515 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.516 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.516 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:44.516 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.210273+0000", 2024-04-19T02:03:44.516 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.4513999999999998e-05 2024-04-19T02:03:44.516 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.516 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.516 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:44.516 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.211091+0000", 2024-04-19T02:03:44.516 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00081805699999999996 2024-04-19T02:03:44.516 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.516 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.516 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:44.516 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.212584+0000", 2024-04-19T02:03:44.517 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0014929260000000001 2024-04-19T02:03:44.517 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.517 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.517 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:44.517 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.216825+0000", 2024-04-19T02:03:44.517 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.004240466 2024-04-19T02:03:44.517 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.517 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.517 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:44.517 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.216912+0000", 2024-04-19T02:03:44.517 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7736000000000002e-05 2024-04-19T02:03:44.517 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.517 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:44.517 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.518 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.518 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.518 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:482 3.4s0 3:2859204c:::smithi05930959-39:head [append 0~622592] snapc a=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:03:44.518 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:31.213688+0000", 2024-04-19T02:03:44.518 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 13.215318555, 2024-04-19T02:03:44.518 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0073709329999999997, 2024-04-19T02:03:44.518 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:44.518 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:44.518 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:44.518 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:44.518 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:44.518 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:44.518 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 482 2024-04-19T02:03:44.519 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.519 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:44.519 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.519 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:44.519 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.213688+0000", 2024-04-19T02:03:44.519 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.519 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.519 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.519 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:44.519 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.213688+0000", 2024-04-19T02:03:44.519 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.519 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.519 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.520 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:44.520 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.213689+0000", 2024-04-19T02:03:44.520 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8080000000000001e-06 2024-04-19T02:03:44.520 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.520 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.520 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:44.520 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.214086+0000", 2024-04-19T02:03:44.520 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000396184 2024-04-19T02:03:44.520 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.520 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.520 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:44.520 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.214101+0000", 2024-04-19T02:03:44.520 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4994000000000001e-05 2024-04-19T02:03:44.521 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.521 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.521 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:44.521 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.214109+0000", 2024-04-19T02:03:44.521 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5730000000000008e-06 2024-04-19T02:03:44.521 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.521 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.521 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:44.521 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.214352+0000", 2024-04-19T02:03:44.521 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024302300000000001 2024-04-19T02:03:44.521 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.521 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.522 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:44.522 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.214451+0000", 2024-04-19T02:03:44.522 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9166999999999994e-05 2024-04-19T02:03:44.522 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.522 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.522 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:44.522 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.215665+0000", 2024-04-19T02:03:44.522 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0012132849999999999 2024-04-19T02:03:44.522 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.522 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.522 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:44.522 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.218777+0000", 2024-04-19T02:03:44.522 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0031121619999999999 2024-04-19T02:03:44.523 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.523 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.523 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:44.523 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.220970+0000", 2024-04-19T02:03:44.523 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0021936030000000001 2024-04-19T02:03:44.523 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.523 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.523 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:44.523 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.221059+0000", 2024-04-19T02:03:44.523 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.8133999999999993e-05 2024-04-19T02:03:44.523 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.523 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:44.523 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.524 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.524 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.524 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:483 3.4s0 3:2859204c:::smithi05930959-39:head [setxattr _header (56) in=7b] snapc a=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:03:44.524 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:31.214133+0000", 2024-04-19T02:03:44.524 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 13.214873634, 2024-04-19T02:03:44.524 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0073982509999999998, 2024-04-19T02:03:44.524 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:44.524 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:44.524 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:44.524 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:44.524 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:44.524 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:44.524 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 483 2024-04-19T02:03:44.525 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.525 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:44.525 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.525 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:44.525 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.214133+0000", 2024-04-19T02:03:44.525 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.525 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.525 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.525 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:44.525 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.214133+0000", 2024-04-19T02:03:44.525 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.525 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.525 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.526 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:44.526 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.214133+0000", 2024-04-19T02:03:44.526 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.0699999999999996e-07 2024-04-19T02:03:44.526 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.526 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.526 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:44.526 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.214139+0000", 2024-04-19T02:03:44.526 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.4229999999999996e-06 2024-04-19T02:03:44.526 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.526 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.526 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:44.526 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.214148+0000", 2024-04-19T02:03:44.526 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.5540000000000003e-06 2024-04-19T02:03:44.527 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.527 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.527 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:44.527 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.214154+0000", 2024-04-19T02:03:44.527 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.2439999999999998e-06 2024-04-19T02:03:44.527 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.527 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.527 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:44.527 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.216962+0000", 2024-04-19T02:03:44.527 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002807866 2024-04-19T02:03:44.527 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.527 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.527 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:44.528 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.217064+0000", 2024-04-19T02:03:44.528 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000101216 2024-04-19T02:03:44.528 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.528 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.528 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:44.528 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.217542+0000", 2024-04-19T02:03:44.528 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00047826900000000002 2024-04-19T02:03:44.528 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.528 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.528 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:44.528 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.219838+0000", 2024-04-19T02:03:44.528 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002296574 2024-04-19T02:03:44.528 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.529 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.529 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:44.529 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.221438+0000", 2024-04-19T02:03:44.529 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0015996890000000001 2024-04-19T02:03:44.529 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.529 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.529 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:44.529 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.221531+0000", 2024-04-19T02:03:44.529 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.2708999999999996e-05 2024-04-19T02:03:44.529 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.529 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:44.529 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.529 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.530 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.530 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:484 3.4s0 3:2859204c:::smithi05930959-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:03:44.530 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:31.214166+0000", 2024-04-19T02:03:44.530 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 13.21484023, 2024-04-19T02:03:44.530 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010260811999999999, 2024-04-19T02:03:44.530 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:44.530 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:44.530 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:44.530 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:44.530 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:44.530 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:44.530 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 484 2024-04-19T02:03:44.530 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.531 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:44.531 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.531 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:44.531 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.214166+0000", 2024-04-19T02:03:44.531 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.531 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.531 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.531 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:44.531 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.214166+0000", 2024-04-19T02:03:44.531 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.531 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.531 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.531 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:44.532 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.214167+0000", 2024-04-19T02:03:44.532 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7000000000000005e-07 2024-04-19T02:03:44.532 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.532 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.532 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:44.532 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.214169+0000", 2024-04-19T02:03:44.532 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6240000000000002e-06 2024-04-19T02:03:44.532 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.532 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.532 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:44.532 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.214178+0000", 2024-04-19T02:03:44.532 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.721e-06 2024-04-19T02:03:44.532 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.532 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.533 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:44.533 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.214187+0000", 2024-04-19T02:03:44.533 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.5710000000000004e-06 2024-04-19T02:03:44.533 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.533 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.533 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:44.533 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.217834+0000", 2024-04-19T02:03:44.533 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.003646846 2024-04-19T02:03:44.533 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.533 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.533 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:44.533 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.217931+0000", 2024-04-19T02:03:44.533 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.6584999999999998e-05 2024-04-19T02:03:44.534 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.534 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.534 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:44.534 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.222283+0000", 2024-04-19T02:03:44.534 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0043518469999999998 2024-04-19T02:03:44.534 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.534 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.534 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:44.534 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.222384+0000", 2024-04-19T02:03:44.534 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000101426 2024-04-19T02:03:44.534 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.534 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.534 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:44.535 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.224427+0000", 2024-04-19T02:03:44.535 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002042622 2024-04-19T02:03:44.535 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.535 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:44.535 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.535 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.535 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.535 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.8s1 57/26 ECSubWrite(tid=120, reqid=osd.4.0:120, at_version=57'63, trim_to=0'0, roll_forward_to=56'61))", 2024-04-19T02:03:44.535 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:41.151500+0000", 2024-04-19T02:03:44.535 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 3.2775063740000001, 2024-04-19T02:03:44.535 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.007592934, 2024-04-19T02:03:44.535 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:44.535 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:44.536 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:44.536 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:44.536 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.536 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:44.536 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.151500+0000", 2024-04-19T02:03:44.536 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.536 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.536 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.536 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:44.536 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.151500+0000", 2024-04-19T02:03:44.536 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:44.536 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.536 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.537 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:44.537 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.151500+0000", 2024-04-19T02:03:44.537 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.03e-07 2024-04-19T02:03:44.537 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.537 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.537 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:44.537 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.151537+0000", 2024-04-19T02:03:44.537 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.7073999999999998e-05 2024-04-19T02:03:44.537 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.537 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.537 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:44.537 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.151552+0000", 2024-04-19T02:03:44.537 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5129e-05 2024-04-19T02:03:44.537 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.538 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.538 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:44.538 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.151564+0000", 2024-04-19T02:03:44.538 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1894e-05 2024-04-19T02:03:44.538 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.538 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.538 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:44.538 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.151654+0000", 2024-04-19T02:03:44.538 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.9244000000000001e-05 2024-04-19T02:03:44.538 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.538 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.538 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:44.539 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.151670+0000", 2024-04-19T02:03:44.539 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6341000000000002e-05 2024-04-19T02:03:44.539 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.539 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.539 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:44.539 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.159033+0000", 2024-04-19T02:03:44.539 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0073635310000000004 2024-04-19T02:03:44.539 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:44.539 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:44.539 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:44.539 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.159093+0000", 2024-04-19T02:03:44.539 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.9317999999999997e-05 2024-04-19T02:03:44.539 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.539 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:44.540 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.540 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:44.540 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:44.540 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:44.542 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-19T02:03:44.542 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:44.542+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:44.642 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:44.641+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:44.716 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:44.716 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:03:44.716 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-04-19T02:03:44.716 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:44.730 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-19T02:03:44.775 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:44.778+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:44.844 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:44.846+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:44.852 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:44.852 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:03:44.852 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-04-19T02:03:44.852 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-04-19T02:03:44.852 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:44.866 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-19T02:03:44.945 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:44.945+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:45.042 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:45.042 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-04-19T02:03:45.042 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-04-19T02:03:45.042 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-04-19T02:03:45.042 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.043 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 40/26 ECSubWrite(tid=30, reqid=client.4509.0:359, at_version=40'16, trim_to=0'0, roll_forward_to=39'14))", 2024-04-19T02:03:45.043 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.790646+0000", 2024-04-19T02:03:45.043 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 20.251823365, 2024-04-19T02:03:45.043 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.004294422, 2024-04-19T02:03:45.043 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.043 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.043 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:45.043 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.043 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.043 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.043 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.790646+0000", 2024-04-19T02:03:45.043 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.043 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.044 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.044 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.044 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.790646+0000", 2024-04-19T02:03:45.044 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.044 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.044 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.044 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.044 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.790647+0000", 2024-04-19T02:03:45.044 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7400000000000003e-07 2024-04-19T02:03:45.044 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.044 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.044 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.044 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.791098+0000", 2024-04-19T02:03:45.044 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00045124599999999998 2024-04-19T02:03:45.045 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.045 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.045 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.045 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.791116+0000", 2024-04-19T02:03:45.045 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7351999999999999e-05 2024-04-19T02:03:45.045 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.045 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.045 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.045 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.791126+0000", 2024-04-19T02:03:45.045 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0606e-05 2024-04-19T02:03:45.045 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.045 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.045 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.046 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.791206+0000", 2024-04-19T02:03:45.046 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0018999999999994e-05 2024-04-19T02:03:45.046 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.046 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.046 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:45.046 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.791217+0000", 2024-04-19T02:03:45.046 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0361e-05 2024-04-19T02:03:45.046 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.046 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.046 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:45.046 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.794897+0000", 2024-04-19T02:03:45.046 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0036802810000000001 2024-04-19T02:03:45.046 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.047 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.047 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.047 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.794941+0000", 2024-04-19T02:03:45.047 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.3983000000000002e-05 2024-04-19T02:03:45.047 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.047 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.047 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.047 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.047 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.047 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s5 42/26 ECSubWrite(tid=43, reqid=osd.1.0:43, at_version=42'30, trim_to=0'0, roll_forward_to=40'28))", 2024-04-19T02:03:45.047 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:26.737676+0000", 2024-04-19T02:03:45.047 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 18.304793701000001, 2024-04-19T02:03:45.047 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0079882579999999998, 2024-04-19T02:03:45.047 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.048 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.048 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:45.048 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.048 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.048 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.048 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.737676+0000", 2024-04-19T02:03:45.048 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.048 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.048 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.048 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.048 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.737676+0000", 2024-04-19T02:03:45.048 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.048 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.048 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.048 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.049 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.737676+0000", 2024-04-19T02:03:45.049 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3999999999999997e-07 2024-04-19T02:03:45.049 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.049 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.049 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.049 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.737712+0000", 2024-04-19T02:03:45.049 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5405999999999999e-05 2024-04-19T02:03:45.049 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.049 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.049 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.049 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.737731+0000", 2024-04-19T02:03:45.049 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9437e-05 2024-04-19T02:03:45.049 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.049 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.049 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.049 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.737746+0000", 2024-04-19T02:03:45.050 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5013e-05 2024-04-19T02:03:45.050 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.050 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.050 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.050 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.737887+0000", 2024-04-19T02:03:45.050 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014084300000000001 2024-04-19T02:03:45.050 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.050 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.050 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:45.050 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.737902+0000", 2024-04-19T02:03:45.050 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5138e-05 2024-04-19T02:03:45.050 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.050 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.050 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:45.051 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.745612+0000", 2024-04-19T02:03:45.051 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0077100049999999998 2024-04-19T02:03:45.051 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.051 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.051 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.051 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.745664+0000", 2024-04-19T02:03:45.051 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.2076000000000001e-05 2024-04-19T02:03:45.051 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.051 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.051 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.051 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.051 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.051 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s5 42/26 ECSubWrite(tid=44, reqid=osd.1.0:44, at_version=42'32, trim_to=0'0, roll_forward_to=40'28))", 2024-04-19T02:03:45.051 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:26.739008+0000", 2024-04-19T02:03:45.051 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 18.303462094, 2024-04-19T02:03:45.052 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0066868379999999996, 2024-04-19T02:03:45.052 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.052 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.052 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:45.052 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.052 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.052 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.052 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.739008+0000", 2024-04-19T02:03:45.052 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.052 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.052 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.052 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.052 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.739008+0000", 2024-04-19T02:03:45.052 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.053 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.053 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.053 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.053 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.739009+0000", 2024-04-19T02:03:45.053 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.4300000000000001e-07 2024-04-19T02:03:45.053 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.053 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.053 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.053 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.739041+0000", 2024-04-19T02:03:45.053 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2642999999999999e-05 2024-04-19T02:03:45.053 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.053 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.053 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.053 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.739057+0000", 2024-04-19T02:03:45.053 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6124e-05 2024-04-19T02:03:45.053 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.054 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.054 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.054 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.739067+0000", 2024-04-19T02:03:45.054 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3389999999999999e-06 2024-04-19T02:03:45.054 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.054 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.054 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.054 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.739141+0000", 2024-04-19T02:03:45.054 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.3794999999999994e-05 2024-04-19T02:03:45.054 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.054 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.054 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:45.054 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.739152+0000", 2024-04-19T02:03:45.054 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0954e-05 2024-04-19T02:03:45.055 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.055 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.055 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:45.055 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.745671+0000", 2024-04-19T02:03:45.055 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0065197450000000004 2024-04-19T02:03:45.055 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.055 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.055 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.055 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.745695+0000", 2024-04-19T02:03:45.055 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3295e-05 2024-04-19T02:03:45.055 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.055 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.055 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.055 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.056 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.056 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.5s2 42/26 ECSubWrite(tid=51, reqid=osd.10.0:51, at_version=42'27, trim_to=0'0, roll_forward_to=39'25))", 2024-04-19T02:03:45.056 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:26.748457+0000", 2024-04-19T02:03:45.056 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 18.294012725999998, 2024-04-19T02:03:45.056 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0043907169999999997, 2024-04-19T02:03:45.056 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.056 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.056 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:45.056 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.056 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.056 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.056 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.748457+0000", 2024-04-19T02:03:45.057 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.057 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.057 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.057 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.057 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.748457+0000", 2024-04-19T02:03:45.057 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.057 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.057 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.057 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.057 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.748458+0000", 2024-04-19T02:03:45.057 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0699999999999998e-07 2024-04-19T02:03:45.057 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.057 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.057 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.057 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.748491+0000", 2024-04-19T02:03:45.057 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.392e-05 2024-04-19T02:03:45.058 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.058 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.058 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.058 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.748509+0000", 2024-04-19T02:03:45.058 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7450000000000001e-05 2024-04-19T02:03:45.058 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.058 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.058 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.058 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.748521+0000", 2024-04-19T02:03:45.058 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2591e-05 2024-04-19T02:03:45.058 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.058 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.058 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.059 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.748598+0000", 2024-04-19T02:03:45.059 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6886999999999998e-05 2024-04-19T02:03:45.059 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.059 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.059 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:45.059 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.748608+0000", 2024-04-19T02:03:45.059 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.6709999999999994e-06 2024-04-19T02:03:45.059 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.060 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.060 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:45.060 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.752792+0000", 2024-04-19T02:03:45.060 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0041843339999999996 2024-04-19T02:03:45.060 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.060 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.060 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.060 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.752848+0000", 2024-04-19T02:03:45.060 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5457000000000001e-05 2024-04-19T02:03:45.060 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.060 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.060 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.061 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.061 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.061 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.as7 42/26 ECSubWrite(tid=82, reqid=osd.11.0:82, at_version=42'34, trim_to=0'0, roll_forward_to=39'32))", 2024-04-19T02:03:45.061 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:26.757483+0000", 2024-04-19T02:03:45.061 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 18.284987092000001, 2024-04-19T02:03:45.061 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0054335750000000004, 2024-04-19T02:03:45.061 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.061 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.061 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:45.061 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.061 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.061 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.061 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757483+0000", 2024-04-19T02:03:45.061 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.062 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.062 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.062 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.062 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757483+0000", 2024-04-19T02:03:45.062 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.062 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.062 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.062 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.062 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757483+0000", 2024-04-19T02:03:45.062 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1e-07 2024-04-19T02:03:45.062 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.062 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.062 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.063 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757508+0000", 2024-04-19T02:03:45.063 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4681999999999998e-05 2024-04-19T02:03:45.063 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.063 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.063 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.063 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757525+0000", 2024-04-19T02:03:45.063 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6869000000000001e-05 2024-04-19T02:03:45.063 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.063 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.063 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.063 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757535+0000", 2024-04-19T02:03:45.063 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0074999999999999e-05 2024-04-19T02:03:45.063 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.063 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.064 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.064 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757714+0000", 2024-04-19T02:03:45.064 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000179388 2024-04-19T02:03:45.064 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.064 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.064 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:45.064 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757729+0000", 2024-04-19T02:03:45.064 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4703e-05 2024-04-19T02:03:45.064 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.064 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.064 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:45.064 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.762889+0000", 2024-04-19T02:03:45.064 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0051599430000000002 2024-04-19T02:03:45.064 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.065 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.065 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.065 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.762916+0000", 2024-04-19T02:03:45.065 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7605000000000001e-05 2024-04-19T02:03:45.065 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.065 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.065 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.065 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.065 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.065 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs2 42/26 ECSubWrite(tid=83, reqid=osd.11.0:83, at_version=42'26, trim_to=0'0, roll_forward_to=39'24))", 2024-04-19T02:03:45.065 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:26.757561+0000", 2024-04-19T02:03:45.065 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 18.284909213999999, 2024-04-19T02:03:45.065 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0053118139999999998, 2024-04-19T02:03:45.065 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.066 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.066 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:45.066 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.066 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.066 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.066 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757561+0000", 2024-04-19T02:03:45.066 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.066 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.066 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.066 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.066 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757561+0000", 2024-04-19T02:03:45.066 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.066 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.066 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.066 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.067 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757561+0000", 2024-04-19T02:03:45.067 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.58e-07 2024-04-19T02:03:45.067 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.067 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.067 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.067 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757602+0000", 2024-04-19T02:03:45.067 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0790000000000001e-05 2024-04-19T02:03:45.067 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.067 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.067 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.067 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757612+0000", 2024-04-19T02:03:45.067 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0078000000000001e-05 2024-04-19T02:03:45.067 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.067 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.067 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.068 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757648+0000", 2024-04-19T02:03:45.068 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6046e-05 2024-04-19T02:03:45.068 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.068 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.068 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.068 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757718+0000", 2024-04-19T02:03:45.068 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9772999999999998e-05 2024-04-19T02:03:45.068 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.068 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.068 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:45.068 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757727+0000", 2024-04-19T02:03:45.068 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.7280000000000007e-06 2024-04-19T02:03:45.068 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.068 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.068 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:45.068 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.762829+0000", 2024-04-19T02:03:45.069 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0051016539999999997 2024-04-19T02:03:45.069 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.069 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.069 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.069 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.762872+0000", 2024-04-19T02:03:45.069 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.3387000000000003e-05 2024-04-19T02:03:45.069 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.069 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.069 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.069 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.069 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.069 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.5s2 42/26 ECSubWrite(tid=52, reqid=osd.10.0:52, at_version=42'29, trim_to=0'0, roll_forward_to=39'25))", 2024-04-19T02:03:45.069 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:26.759271+0000", 2024-04-19T02:03:45.069 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 18.283199191000001, 2024-04-19T02:03:45.069 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0041464099999999997, 2024-04-19T02:03:45.070 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.070 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.070 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:45.070 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.070 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.070 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.070 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.759271+0000", 2024-04-19T02:03:45.070 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.070 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.070 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.070 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.070 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.759271+0000", 2024-04-19T02:03:45.070 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.071 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.071 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.071 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.071 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.759272+0000", 2024-04-19T02:03:45.071 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2240000000000001e-06 2024-04-19T02:03:45.071 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.071 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.071 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.071 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.759296+0000", 2024-04-19T02:03:45.071 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4555000000000001e-05 2024-04-19T02:03:45.071 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.071 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.071 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.071 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.759317+0000", 2024-04-19T02:03:45.071 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0384999999999999e-05 2024-04-19T02:03:45.072 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.072 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.072 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.072 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.759326+0000", 2024-04-19T02:03:45.072 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7819999999999999e-06 2024-04-19T02:03:45.072 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.072 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.072 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.072 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.759406+0000", 2024-04-19T02:03:45.072 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0458999999999997e-05 2024-04-19T02:03:45.072 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.072 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.072 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:45.072 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.759419+0000", 2024-04-19T02:03:45.073 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3050999999999999e-05 2024-04-19T02:03:45.073 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.073 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.073 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:45.073 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.763377+0000", 2024-04-19T02:03:45.073 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0039574299999999996 2024-04-19T02:03:45.073 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.073 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.073 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.073 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.763417+0000", 2024-04-19T02:03:45.073 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0524e-05 2024-04-19T02:03:45.073 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.073 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.073 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.073 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.073 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.074 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 42/26 ECSubWrite(tid=32, reqid=osd.9.0:32, at_version=42'18, trim_to=0'0, roll_forward_to=40'16))", 2024-04-19T02:03:45.074 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:26.765866+0000", 2024-04-19T02:03:45.074 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 18.276603598000001, 2024-04-19T02:03:45.074 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0081652840000000001, 2024-04-19T02:03:45.074 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.074 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.074 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:45.074 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.074 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.074 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.074 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.765866+0000", 2024-04-19T02:03:45.074 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.074 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.074 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.075 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.075 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.765866+0000", 2024-04-19T02:03:45.075 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.075 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.075 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.075 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.075 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.765867+0000", 2024-04-19T02:03:45.075 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7599999999999998e-07 2024-04-19T02:03:45.075 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.075 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.075 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.075 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.765890+0000", 2024-04-19T02:03:45.075 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3459e-05 2024-04-19T02:03:45.075 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.075 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.076 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.076 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.765901+0000", 2024-04-19T02:03:45.076 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1426e-05 2024-04-19T02:03:45.076 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.076 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.076 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.076 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.765911+0000", 2024-04-19T02:03:45.076 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.2969999999999992e-06 2024-04-19T02:03:45.076 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.076 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.076 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.076 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.765996+0000", 2024-04-19T02:03:45.076 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5568999999999993e-05 2024-04-19T02:03:45.076 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.077 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.077 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:45.077 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.766011+0000", 2024-04-19T02:03:45.077 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4615e-05 2024-04-19T02:03:45.077 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.077 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.077 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:45.077 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.773989+0000", 2024-04-19T02:03:45.077 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0079778669999999996 2024-04-19T02:03:45.077 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.077 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.077 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.077 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.774032+0000", 2024-04-19T02:03:45.077 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2775000000000003e-05 2024-04-19T02:03:45.077 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.078 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.078 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.078 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.078 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.078 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 42/26 ECSubWrite(tid=33, reqid=osd.9.0:33, at_version=42'20, trim_to=0'0, roll_forward_to=40'16))", 2024-04-19T02:03:45.078 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:26.770521+0000", 2024-04-19T02:03:45.078 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 18.271949017000001, 2024-04-19T02:03:45.078 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.003789569, 2024-04-19T02:03:45.078 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.078 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.078 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:45.078 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.078 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.078 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.078 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.770521+0000", 2024-04-19T02:03:45.079 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.079 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.079 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.079 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.079 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.770521+0000", 2024-04-19T02:03:45.079 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.079 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.079 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.079 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.079 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.770522+0000", 2024-04-19T02:03:45.079 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.02e-06 2024-04-19T02:03:45.080 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.080 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.080 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.080 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.770547+0000", 2024-04-19T02:03:45.080 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5182e-05 2024-04-19T02:03:45.080 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.080 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.080 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.080 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.770561+0000", 2024-04-19T02:03:45.080 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4083000000000001e-05 2024-04-19T02:03:45.080 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.080 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.080 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.080 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.770571+0000", 2024-04-19T02:03:45.080 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0152999999999999e-05 2024-04-19T02:03:45.081 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.081 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.081 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.081 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.770660+0000", 2024-04-19T02:03:45.081 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.8271999999999998e-05 2024-04-19T02:03:45.081 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.081 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.081 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:45.081 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.770672+0000", 2024-04-19T02:03:45.081 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2581e-05 2024-04-19T02:03:45.081 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.081 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.081 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:45.081 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.774258+0000", 2024-04-19T02:03:45.081 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0035857419999999998 2024-04-19T02:03:45.082 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.082 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.082 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.082 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.774310+0000", 2024-04-19T02:03:45.082 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.2536e-05 2024-04-19T02:03:45.082 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.082 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.082 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.082 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.082 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.082 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 46/26 ECSubWrite(tid=44, reqid=osd.9.0:44, at_version=46'26, trim_to=0'0, roll_forward_to=45'24))", 2024-04-19T02:03:45.082 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.065734+0000", 2024-04-19T02:03:45.082 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 14.976735829999999, 2024-04-19T02:03:45.082 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0090304430000000008, 2024-04-19T02:03:45.083 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.083 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.083 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:45.083 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.083 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.083 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.083 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.065734+0000", 2024-04-19T02:03:45.083 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.083 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.083 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.083 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.083 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.065734+0000", 2024-04-19T02:03:45.083 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.083 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.084 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.084 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.084 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.065734+0000", 2024-04-19T02:03:45.084 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2399999999999999e-07 2024-04-19T02:03:45.084 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.084 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.084 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.084 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.065763+0000", 2024-04-19T02:03:45.084 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8393000000000001e-05 2024-04-19T02:03:45.084 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.084 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.084 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.084 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.065786+0000", 2024-04-19T02:03:45.084 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2909e-05 2024-04-19T02:03:45.085 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.085 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.085 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.085 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.065799+0000", 2024-04-19T02:03:45.085 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2975e-05 2024-04-19T02:03:45.085 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.085 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.085 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.085 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.069211+0000", 2024-04-19T02:03:45.085 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0034118899999999999 2024-04-19T02:03:45.085 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.085 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.085 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:45.085 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.069234+0000", 2024-04-19T02:03:45.085 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.372e-05 2024-04-19T02:03:45.086 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.086 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.086 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:45.086 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.074695+0000", 2024-04-19T02:03:45.086 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0054611779999999997 2024-04-19T02:03:45.086 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.086 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.086 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.086 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.074764+0000", 2024-04-19T02:03:45.086 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8954000000000004e-05 2024-04-19T02:03:45.086 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.086 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.086 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.086 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.086 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.087 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.as7 46/26 ECSubWrite(tid=103, reqid=osd.11.0:103, at_version=46'49, trim_to=0'0, roll_forward_to=45'45))", 2024-04-19T02:03:45.087 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.070195+0000", 2024-04-19T02:03:45.087 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 14.972274933, 2024-04-19T02:03:45.087 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0048631990000000003, 2024-04-19T02:03:45.087 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.087 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.087 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:45.087 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.087 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.087 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.087 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.070195+0000", 2024-04-19T02:03:45.087 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.087 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.087 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.087 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.088 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.070195+0000", 2024-04-19T02:03:45.088 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.088 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.088 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.088 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.088 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.070196+0000", 2024-04-19T02:03:45.088 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8899999999999999e-07 2024-04-19T02:03:45.088 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.088 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.088 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.088 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.070219+0000", 2024-04-19T02:03:45.088 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3485999999999998e-05 2024-04-19T02:03:45.088 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.088 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.088 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.089 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.070230+0000", 2024-04-19T02:03:45.089 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1039000000000001e-05 2024-04-19T02:03:45.089 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.089 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.089 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.089 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.070241+0000", 2024-04-19T02:03:45.089 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0866999999999999e-05 2024-04-19T02:03:45.089 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.089 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.089 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.089 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.070720+0000", 2024-04-19T02:03:45.089 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00047945099999999999 2024-04-19T02:03:45.089 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.089 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.090 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:45.090 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.070734+0000", 2024-04-19T02:03:45.090 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3811e-05 2024-04-19T02:03:45.090 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.090 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.090 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:45.090 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.075014+0000", 2024-04-19T02:03:45.090 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.004280196 2024-04-19T02:03:45.090 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.090 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.090 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.090 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.075058+0000", 2024-04-19T02:03:45.090 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.3659999999999999e-05 2024-04-19T02:03:45.091 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.091 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.091 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.091 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.091 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.091 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s5 46/26 ECSubWrite(tid=59, reqid=osd.1.0:58, at_version=0'0, trim_to=0'0, roll_forward_to=46'44))", 2024-04-19T02:03:45.091 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.077390+0000", 2024-04-19T02:03:45.091 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 14.965079427999999, 2024-04-19T02:03:45.091 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0043230480000000003, 2024-04-19T02:03:45.091 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.091 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.091 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:45.091 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.091 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.092 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.092 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077390+0000", 2024-04-19T02:03:45.092 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.092 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.092 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.092 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.092 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077390+0000", 2024-04-19T02:03:45.092 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.092 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.092 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.092 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.092 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077392+0000", 2024-04-19T02:03:45.093 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2270000000000001e-06 2024-04-19T02:03:45.093 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.093 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.093 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.093 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077409+0000", 2024-04-19T02:03:45.093 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7748999999999999e-05 2024-04-19T02:03:45.093 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.093 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.093 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.093 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077436+0000", 2024-04-19T02:03:45.093 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6451000000000001e-05 2024-04-19T02:03:45.093 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.093 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.093 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.094 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077502+0000", 2024-04-19T02:03:45.094 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.6493000000000002e-05 2024-04-19T02:03:45.094 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.094 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.094 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.094 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077638+0000", 2024-04-19T02:03:45.094 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013563599999999999 2024-04-19T02:03:45.094 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.094 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.094 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:45.094 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077650+0000", 2024-04-19T02:03:45.094 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1569999999999999e-05 2024-04-19T02:03:45.094 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.095 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.095 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:45.095 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.081644+0000", 2024-04-19T02:03:45.095 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0039942950000000001 2024-04-19T02:03:45.095 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.095 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.095 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.095 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.081713+0000", 2024-04-19T02:03:45.095 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9627000000000003e-05 2024-04-19T02:03:45.095 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.095 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.095 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.095 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.095 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.096 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es5 46/26 ECSubWrite(tid=39, reqid=client.4509.0:420, at_version=46'31, trim_to=0'0, roll_forward_to=46'29))", 2024-04-19T02:03:45.096 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.240512+0000", 2024-04-19T02:03:45.096 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 14.801957870000001, 2024-04-19T02:03:45.096 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0043966530000000004, 2024-04-19T02:03:45.096 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.096 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.096 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:45.096 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.096 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.096 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.096 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.240512+0000", 2024-04-19T02:03:45.096 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.096 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.096 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.097 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.097 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.240512+0000", 2024-04-19T02:03:45.097 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.097 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.097 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.097 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.097 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.240514+0000", 2024-04-19T02:03:45.097 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.717e-06 2024-04-19T02:03:45.097 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.097 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.097 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.097 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.242558+0000", 2024-04-19T02:03:45.097 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0020443940000000002 2024-04-19T02:03:45.097 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.097 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.098 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.098 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.242576+0000", 2024-04-19T02:03:45.098 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8127000000000001e-05 2024-04-19T02:03:45.098 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.098 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.098 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.098 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.242585+0000", 2024-04-19T02:03:45.098 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.1300000000000007e-06 2024-04-19T02:03:45.098 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.098 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.098 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.099 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.242678+0000", 2024-04-19T02:03:45.099 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.2808999999999999e-05 2024-04-19T02:03:45.099 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.099 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.099 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:45.099 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.242685+0000", 2024-04-19T02:03:45.099 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.7700000000000004e-06 2024-04-19T02:03:45.099 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.099 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.099 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:45.099 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.244876+0000", 2024-04-19T02:03:45.099 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0021906270000000001 2024-04-19T02:03:45.099 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.099 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.100 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.100 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.244909+0000", 2024-04-19T02:03:45.100 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3079e-05 2024-04-19T02:03:45.100 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.100 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.100 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.100 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.100 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.100 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs2 46/26 ECSubWrite(tid=108, reqid=client.4509.0:448, at_version=46'30, trim_to=0'0, roll_forward_to=45'28))", 2024-04-19T02:03:45.100 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.522984+0000", 2024-04-19T02:03:45.100 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 14.519486317, 2024-04-19T02:03:45.100 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0039496239999999997, 2024-04-19T02:03:45.100 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.100 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.100 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:45.101 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.101 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.101 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.101 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.522984+0000", 2024-04-19T02:03:45.101 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.101 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.101 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.101 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.101 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.522984+0000", 2024-04-19T02:03:45.101 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.101 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.101 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.101 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.101 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.522985+0000", 2024-04-19T02:03:45.102 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.023e-06 2024-04-19T02:03:45.102 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.102 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.102 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.102 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.523423+0000", 2024-04-19T02:03:45.102 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00043817200000000002 2024-04-19T02:03:45.102 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.102 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.102 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.102 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.523443+0000", 2024-04-19T02:03:45.102 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0333999999999999e-05 2024-04-19T02:03:45.102 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.102 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.102 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.103 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.523454+0000", 2024-04-19T02:03:45.103 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0688e-05 2024-04-19T02:03:45.103 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.103 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.103 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.103 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.523735+0000", 2024-04-19T02:03:45.103 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000281379 2024-04-19T02:03:45.103 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.103 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.103 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:45.103 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.523767+0000", 2024-04-19T02:03:45.103 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2048999999999998e-05 2024-04-19T02:03:45.103 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.103 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.103 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:45.103 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.526888+0000", 2024-04-19T02:03:45.104 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0031204599999999998 2024-04-19T02:03:45.104 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.104 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.104 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.104 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.526933+0000", 2024-04-19T02:03:45.104 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5519000000000002e-05 2024-04-19T02:03:45.104 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.104 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.104 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.104 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.104 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.104 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s5 47/26 ECSubWrite(tid=62, reqid=client.4509.0:482, at_version=47'47, trim_to=0'0, roll_forward_to=47'45))", 2024-04-19T02:03:45.104 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:31.216102+0000", 2024-04-19T02:03:45.104 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 13.826368249, 2024-04-19T02:03:45.104 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0045766349999999999, 2024-04-19T02:03:45.104 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.104 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.105 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:45.105 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.105 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.105 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.105 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.216102+0000", 2024-04-19T02:03:45.105 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.105 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.105 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.105 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.105 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.216102+0000", 2024-04-19T02:03:45.105 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.105 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.105 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.105 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.105 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.216103+0000", 2024-04-19T02:03:45.105 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.683e-06 2024-04-19T02:03:45.106 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.106 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.106 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.106 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.216227+0000", 2024-04-19T02:03:45.106 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00012341500000000001 2024-04-19T02:03:45.106 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.106 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.106 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.106 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.216252+0000", 2024-04-19T02:03:45.106 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5525999999999999e-05 2024-04-19T02:03:45.106 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.106 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.106 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.107 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.216263+0000", 2024-04-19T02:03:45.107 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1273e-05 2024-04-19T02:03:45.107 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.107 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.107 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.107 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.216367+0000", 2024-04-19T02:03:45.107 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000103427 2024-04-19T02:03:45.107 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.107 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.107 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:45.107 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.216378+0000", 2024-04-19T02:03:45.107 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0684e-05 2024-04-19T02:03:45.107 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.107 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.107 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:45.108 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.220623+0000", 2024-04-19T02:03:45.108 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0042451870000000001 2024-04-19T02:03:45.108 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.108 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.108 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.108 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.220678+0000", 2024-04-19T02:03:45.108 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5439999999999998e-05 2024-04-19T02:03:45.108 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.108 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.108 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.108 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.108 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.108 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 51/26 ECSubWrite(tid=52, reqid=osd.9.0:51, at_version=0'0, trim_to=0'0, roll_forward_to=51'30))", 2024-04-19T02:03:45.108 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:35.116856+0000", 2024-04-19T02:03:45.108 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 9.9256137419999995, 2024-04-19T02:03:45.109 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0055490599999999998, 2024-04-19T02:03:45.109 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.109 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.109 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:45.109 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.109 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.109 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.109 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:35.116856+0000", 2024-04-19T02:03:45.109 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.109 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.109 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.109 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.109 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:35.116856+0000", 2024-04-19T02:03:45.109 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.109 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.110 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.110 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.110 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:35.116857+0000", 2024-04-19T02:03:45.110 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0800000000000004e-07 2024-04-19T02:03:45.110 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.110 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.110 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.110 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:35.116871+0000", 2024-04-19T02:03:45.110 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.456e-05 2024-04-19T02:03:45.110 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.110 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.110 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.110 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:35.116884+0000", 2024-04-19T02:03:45.110 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2374999999999999e-05 2024-04-19T02:03:45.110 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.111 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.111 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.111 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:35.116894+0000", 2024-04-19T02:03:45.111 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.026e-05 2024-04-19T02:03:45.111 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.111 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.111 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.111 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:35.116986+0000", 2024-04-19T02:03:45.111 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.2021999999999994e-05 2024-04-19T02:03:45.111 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.111 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.111 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:45.111 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:35.117001+0000", 2024-04-19T02:03:45.111 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5235999999999999e-05 2024-04-19T02:03:45.112 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.112 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.112 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:45.112 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:35.122349+0000", 2024-04-19T02:03:45.112 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0053481400000000004 2024-04-19T02:03:45.112 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.112 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.112 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.112 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:35.122405+0000", 2024-04-19T02:03:45.112 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5658999999999997e-05 2024-04-19T02:03:45.112 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.112 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.112 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.112 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.113 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.113 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 52/26 ECSubWrite(tid=55, reqid=client.4509.0:531, at_version=52'34, trim_to=0'0, roll_forward_to=51'30))", 2024-04-19T02:03:45.113 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:36.144326+0000", 2024-04-19T02:03:45.113 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 8.898143653, 2024-04-19T02:03:45.113 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.003845036, 2024-04-19T02:03:45.113 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.113 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.113 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:45.113 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.113 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.113 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.113 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:36.144326+0000", 2024-04-19T02:03:45.113 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.113 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.113 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.114 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.114 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:36.144326+0000", 2024-04-19T02:03:45.114 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.114 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.114 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.114 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.114 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:36.144327+0000", 2024-04-19T02:03:45.114 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.6500000000000008e-07 2024-04-19T02:03:45.114 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.114 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.114 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.114 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:36.144648+0000", 2024-04-19T02:03:45.114 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00032119199999999998 2024-04-19T02:03:45.114 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.114 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.115 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.115 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:36.144678+0000", 2024-04-19T02:03:45.115 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9186000000000001e-05 2024-04-19T02:03:45.115 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.115 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.115 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.115 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:36.144691+0000", 2024-04-19T02:03:45.115 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3866000000000001e-05 2024-04-19T02:03:45.115 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.115 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.115 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.115 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:36.144763+0000", 2024-04-19T02:03:45.115 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.1680000000000005e-05 2024-04-19T02:03:45.115 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.115 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.116 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:45.116 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:36.144792+0000", 2024-04-19T02:03:45.116 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9331000000000001e-05 2024-04-19T02:03:45.116 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.116 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.116 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:45.116 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:36.148135+0000", 2024-04-19T02:03:45.116 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0033424140000000001 2024-04-19T02:03:45.116 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.116 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.116 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.116 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:36.148171+0000", 2024-04-19T02:03:45.116 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6402000000000001e-05 2024-04-19T02:03:45.116 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.116 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.116 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.117 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.117 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.117 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s5 57/26 ECSubWrite(tid=71, reqid=osd.1.0:71, at_version=57'56, trim_to=0'0, roll_forward_to=55'54))", 2024-04-19T02:03:45.117 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:41.141947+0000", 2024-04-19T02:03:45.117 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 3.9005223550000001, 2024-04-19T02:03:45.117 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0043941919999999999, 2024-04-19T02:03:45.117 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.117 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.117 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:45.117 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.117 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.117 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.117 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.141947+0000", 2024-04-19T02:03:45.117 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.117 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.117 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.118 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.118 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.141947+0000", 2024-04-19T02:03:45.118 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.118 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.118 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.118 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.118 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.141948+0000", 2024-04-19T02:03:45.118 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.3700000000000001e-07 2024-04-19T02:03:45.118 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.118 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.118 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.118 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.141986+0000", 2024-04-19T02:03:45.118 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.7758e-05 2024-04-19T02:03:45.118 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.118 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.119 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.119 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.142020+0000", 2024-04-19T02:03:45.119 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4146999999999997e-05 2024-04-19T02:03:45.119 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.119 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.119 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.119 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.142036+0000", 2024-04-19T02:03:45.119 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5726000000000002e-05 2024-04-19T02:03:45.119 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.119 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.119 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.119 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.142223+0000", 2024-04-19T02:03:45.119 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00018726600000000001 2024-04-19T02:03:45.119 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.120 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.120 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:45.120 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.142241+0000", 2024-04-19T02:03:45.120 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8413000000000001e-05 2024-04-19T02:03:45.120 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.120 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.120 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:45.120 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.146291+0000", 2024-04-19T02:03:45.120 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0040499999999999998 2024-04-19T02:03:45.120 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.120 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.120 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.120 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.146342+0000", 2024-04-19T02:03:45.120 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.0445000000000002e-05 2024-04-19T02:03:45.121 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.121 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.121 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.121 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.121 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.121 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.as7 57/26 ECSubWrite(tid=136, reqid=osd.11.0:136, at_version=57'62, trim_to=0'0, roll_forward_to=55'60))", 2024-04-19T02:03:45.121 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:41.154457+0000", 2024-04-19T02:03:45.121 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 3.888012759, 2024-04-19T02:03:45.121 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0055113590000000004, 2024-04-19T02:03:45.121 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.121 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.121 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:45.121 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.121 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.121 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.122 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.154457+0000", 2024-04-19T02:03:45.122 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.122 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.122 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.122 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.122 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.154457+0000", 2024-04-19T02:03:45.122 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.122 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.122 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.122 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.122 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.154457+0000", 2024-04-19T02:03:45.122 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2300000000000002e-07 2024-04-19T02:03:45.122 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.122 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.123 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.123 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.154489+0000", 2024-04-19T02:03:45.123 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1301999999999999e-05 2024-04-19T02:03:45.123 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.123 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.123 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.123 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.154506+0000", 2024-04-19T02:03:45.123 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7554000000000002e-05 2024-04-19T02:03:45.123 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.123 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.123 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.123 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.154519+0000", 2024-04-19T02:03:45.123 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3074000000000001e-05 2024-04-19T02:03:45.123 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.124 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.124 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.124 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.154628+0000", 2024-04-19T02:03:45.124 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108974 2024-04-19T02:03:45.124 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.124 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.124 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:45.124 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.154653+0000", 2024-04-19T02:03:45.124 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4207999999999999e-05 2024-04-19T02:03:45.124 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.124 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.124 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:45.124 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.159911+0000", 2024-04-19T02:03:45.124 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0052585059999999996 2024-04-19T02:03:45.124 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.124 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.125 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.125 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.159968+0000", 2024-04-19T02:03:45.125 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7417999999999998e-05 2024-04-19T02:03:45.125 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.125 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.125 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.125 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.125 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.125 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s5 57/26 ECSubWrite(tid=72, reqid=osd.1.0:71, at_version=0'0, trim_to=0'0, roll_forward_to=57'56))", 2024-04-19T02:03:45.125 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:41.156487+0000", 2024-04-19T02:03:45.125 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 3.8859829420000001, 2024-04-19T02:03:45.125 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0058785720000000003, 2024-04-19T02:03:45.125 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.125 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.126 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:45.126 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.126 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.126 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.126 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.156487+0000", 2024-04-19T02:03:45.126 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.126 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.126 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.126 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.126 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.156487+0000", 2024-04-19T02:03:45.126 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.126 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.126 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.126 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.127 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.156488+0000", 2024-04-19T02:03:45.127 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2449999999999999e-06 2024-04-19T02:03:45.127 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.127 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.127 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.127 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.156511+0000", 2024-04-19T02:03:45.127 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2429000000000002e-05 2024-04-19T02:03:45.127 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.127 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.127 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.127 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.156530+0000", 2024-04-19T02:03:45.127 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9389999999999999e-05 2024-04-19T02:03:45.127 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.127 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.127 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.128 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.156543+0000", 2024-04-19T02:03:45.128 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2676e-05 2024-04-19T02:03:45.128 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.128 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.128 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.128 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.156876+0000", 2024-04-19T02:03:45.128 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00033304599999999998 2024-04-19T02:03:45.128 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.128 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.128 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:45.128 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.156891+0000", 2024-04-19T02:03:45.128 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4921e-05 2024-04-19T02:03:45.128 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.128 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.129 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:45.129 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.162307+0000", 2024-04-19T02:03:45.129 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.005416227 2024-04-19T02:03:45.129 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.129 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.129 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.129 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.162365+0000", 2024-04-19T02:03:45.129 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.8638000000000003e-05 2024-04-19T02:03:45.129 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.129 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.129 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.129 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.129 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.129 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:45.131 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-19T02:03:45.132 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:45.134+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:45.145 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-19T02:03:45.145 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-19T02:03:45.146 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2024-04-19T02:03:45.146 INFO:tasks.ceph.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 12 2024-04-19T02:03:45.146 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T02:03:45.166 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is set 2024-04-19T02:03:45.171 INFO:tasks.rados.rados.0.smithi059.stdout:244: done (2 left) 2024-04-19T02:03:45.171 INFO:tasks.rados.rados.0.smithi059.stdout:246: done (1 left) 2024-04-19T02:03:45.171 INFO:tasks.rados.rados.0.smithi059.stdout:248: done (0 left) 2024-04-19T02:03:45.171 INFO:tasks.rados.rados.0.smithi059.stdout:249: rollback oid 33 current snap is 18 2024-04-19T02:03:45.171 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 33 to 14 2024-04-19T02:03:45.171 INFO:tasks.rados.rados.0.smithi059.stdout:250: delete oid 44 current snap is 18 2024-04-19T02:03:45.171 INFO:tasks.rados.rados.0.smithi059.stdout:249: finishing rollback tid 1 to smithi05930959-33 2024-04-19T02:03:45.171 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 33 v 0 (ObjNum 161 snap 0 seq_num 1) dirty dne 2024-04-19T02:03:45.181 INFO:tasks.rados.rados.0.smithi059.stdout:249: done (1 left) 2024-04-19T02:03:45.181 INFO:tasks.rados.rados.0.smithi059.stdout:250: done (0 left) 2024-04-19T02:03:45.181 INFO:tasks.rados.rados.0.smithi059.stdout:251: read oid 40 snap -1 2024-04-19T02:03:45.181 INFO:tasks.rados.rados.0.smithi059.stdout:251: expect (ObjNum 71 snap 5 seq_num 71) 2024-04-19T02:03:45.181 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 8 current snap is 18 2024-04-19T02:03:45.181 INFO:tasks.rados.rados.0.smithi059.stdout:252: seq_num 119 ranges {1097728=458752,1556480=507904,2064384=507904,2572288=458752,3031040=81920} 2024-04-19T02:03:45.199 INFO:tasks.rados.rados.0.smithi059.stdout:252: writing smithi05930959-8 from 1097728 to 1556480 tid 1 2024-04-19T02:03:45.206 INFO:tasks.rados.rados.0.smithi059.stdout:252: writing smithi05930959-8 from 1556480 to 2064384 tid 2 2024-04-19T02:03:45.212 INFO:tasks.rados.rados.0.smithi059.stdout:252: writing smithi05930959-8 from 2064384 to 2572288 tid 3 2024-04-19T02:03:45.217 INFO:tasks.rados.rados.0.smithi059.stdout:252: writing smithi05930959-8 from 2572288 to 3031040 tid 4 2024-04-19T02:03:45.219 INFO:tasks.rados.rados.0.smithi059.stdout:252: writing smithi05930959-8 from 3031040 to 3112960 tid 5 2024-04-19T02:03:45.219 INFO:tasks.rados.rados.0.smithi059.stdout:253: snap_create 2024-04-19T02:03:45.219 INFO:tasks.rados.rados.0.smithi059.stdout:251: expect (ObjNum 71 snap 5 seq_num 71) 2024-04-19T02:03:45.231 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:45.233+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:45.265 INFO:tasks.rados.rados.0.smithi059.stdout:252: finishing write tid 1 to smithi05930959-8 2024-04-19T02:03:45.265 INFO:tasks.rados.rados.0.smithi059.stdout:252: finishing write tid 2 to smithi05930959-8 2024-04-19T02:03:45.265 INFO:tasks.rados.rados.0.smithi059.stdout:252: finishing write tid 3 to smithi05930959-8 2024-04-19T02:03:45.265 INFO:tasks.rados.rados.0.smithi059.stdout:252: finishing write tid 4 to smithi05930959-8 2024-04-19T02:03:45.265 INFO:tasks.rados.rados.0.smithi059.stdout:252: finishing write tid 5 to smithi05930959-8 2024-04-19T02:03:45.265 INFO:tasks.rados.rados.0.smithi059.stdout:252: finishing write tid 6 to smithi05930959-8 2024-04-19T02:03:45.266 INFO:tasks.rados.rados.0.smithi059.stdout:252: finishing write tid 7 to smithi05930959-8 2024-04-19T02:03:45.266 INFO:tasks.rados.rados.0.smithi059.stdout:252: oid 8 updating version 0 to 33 2024-04-19T02:03:45.266 INFO:tasks.rados.rados.0.smithi059.stdout:252: oid 8 updating version 33 to 34 2024-04-19T02:03:45.266 INFO:tasks.rados.rados.0.smithi059.stdout:252: oid 8 version 34 is already newer than 32 2024-04-19T02:03:45.266 INFO:tasks.rados.rados.0.smithi059.stdout:252: oid 8 version 34 is already newer than 29 2024-04-19T02:03:45.266 INFO:tasks.rados.rados.0.smithi059.stdout:252: oid 8 version 34 is already newer than 30 2024-04-19T02:03:45.266 INFO:tasks.rados.rados.0.smithi059.stdout:252: oid 8 version 34 is already newer than 31 2024-04-19T02:03:45.266 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 8 v 34 (ObjNum 119 snap 18 seq_num 119) dirty exists 2024-04-19T02:03:45.266 INFO:tasks.rados.rados.0.smithi059.stdout:252: left oid 8 (ObjNum 119 snap 18 seq_num 119) 2024-04-19T02:03:45.300 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:45.301 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:03:45.301 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-04-19T02:03:45.301 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:45.317 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-19T02:03:45.359 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:45.362+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:45.433 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:45.434+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:45.477 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:03:45.477 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-19T02:03:45.478 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:45.478 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:03:45.478 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-04-19T02:03:45.478 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-04-19T02:03:45.478 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:45.493 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-19T02:03:45.498 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2024-04-19T02:03:45.173475+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":85475872,"num_objects":70,"num_object_clones":23,"num_object_copies":548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":6,"num_read":446,"num_read_kb":107922,"num_write":708,"num_write_kb":105082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":813,"ondisk_log_size":813,"up":146,"acting":146,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1132462080,"kb_used":519072,"kb_used_data":169812,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1131943008,"statfs":{"total":1159641169920,"available":1159109640192,"internally_reserved":0,"allocated":173887488,"data_stored":205180783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9666560,"num_objects":11,"num_object_clones":11,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":3,"num_read":19,"num_read_kb":8592,"num_write":60,"num_write_kb":11072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.030149"},"pg_stats":[{"pgid":"3.f","version":"60'43","reported_seq":224,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-19T02:03:44.197238+0000","last_change":"2024-04-19T02:03:44.202653+0000","last_active":"2024-04-19T02:03:44.197238+0000","last_peered":"2024-04-19T02:03:44.197238+0000","last_clean":"2024-04-19T02:03:44.197238+0000","last_became_active":"2024-04-19T02:03:11.728136+0000","last_became_peered":"2024-04-19T02:03:11.728136+0000","last_unstale":"2024-04-19T02:03:44.197238+0000","last_undegraded":"2024-04-19T02:03:44.197238+0000","last_fullsized":"2024-04-19T02:03:44.197238+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:16.261470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023241949000000001,"stat_sum":{"num_bytes":8192000,"num_objects":4,"num_object_clones":1,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":10858,"num_write":27,"num_write_kb":9008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,2,3,5,8,0,7],"acting":[11,4,2,3,5,8,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"b","length":"1"},{"start":"e","length":"2"},{"start":"11","length":"1"}]},{"pgid":"3.e","version":"55'35","reported_seq":216,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-19T02:03:44.166592+0000","last_change":"2024-04-19T02:03:44.167319+0000","last_active":"2024-04-19T02:03:44.166592+0000","last_peered":"2024-04-19T02:03:44.166592+0000","last_clean":"2024-04-19T02:03:44.166592+0000","last_became_active":"2024-04-19T02:03:11.730135+0000","last_became_peered":"2024-04-19T02:03:11.730135+0000","last_unstale":"2024-04-19T02:03:44.166592+0000","last_undegraded":"2024-04-19T02:03:44.166592+0000","last_fullsized":"2024-04-19T02:03:44.166592+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:18.629434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011328519999999999,"stat_sum":{"num_bytes":2392064,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":7014,"num_write":15,"num_write_kb":3552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5,11,0,2,3,9],"acting":[7,4,5,11,0,2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"b","length":"1"},{"start":"e","length":"2"},{"start":"11","length":"1"}]},{"pgid":"3.d","version":"60'65","reported_seq":245,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-19T02:03:44.176956+0000","last_change":"2024-04-19T02:03:44.180758+0000","last_active":"2024-04-19T02:03:44.176956+0000","last_peered":"2024-04-19T02:03:44.176956+0000","last_clean":"2024-04-19T02:03:44.176956+0000","last_became_active":"2024-04-19T02:03:11.632682+0000","last_became_peered":"2024-04-19T02:03:11.632682+0000","last_unstale":"2024-04-19T02:03:44.176956+0000","last_undegraded":"2024-04-19T02:03:44.176956+0000","last_fullsized":"2024-04-19T02:03:44.176956+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:14.285128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013556976,"stat_sum":{"num_bytes":7880704,"num_objects":8,"num_object_clones":4,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":27,"num_read_kb":5818,"num_write":48,"num_write_kb":10128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7,4,5,3,9,0],"acting":[8,11,7,4,5,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"b","length":"1"},{"start":"e","length":"2"},{"start":"11","length":"1"}]},{"pgid":"3.c","version":"60'51","reported_seq":244,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-19T02:03:44.197997+0000","last_change":"2024-04-19T02:03:44.201657+0000","last_active":"2024-04-19T02:03:44.197997+0000","last_peered":"2024-04-19T02:03:44.197997+0000","last_clean":"2024-04-19T02:03:44.197997+0000","last_became_active":"2024-04-19T02:03:11.731568+0000","last_became_peered":"2024-04-19T02:03:11.731568+0000","last_unstale":"2024-04-19T02:03:44.197997+0000","last_undegraded":"2024-04-19T02:03:44.197997+0000","last_fullsized":"2024-04-19T02:03:44.197997+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:13.499257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027117568000000002,"stat_sum":{"num_bytes":7323648,"num_objects":7,"num_object_clones":3,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":14750,"num_write":26,"num_write_kb":8656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,11,7,4,2,0,5],"acting":[9,3,11,7,4,2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"b","length":"1"},{"start":"e","length":"2"},{"start":"11","length":"1"}]},{"pgid":"3.b","version":"47'38","reported_seq":222,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-19T02:03:44.173296+0000","last_change":"2024-04-19T02:03:44.173657+0000","last_active":"2024-04-19T02:03:44.173296+0000","last_peered":"2024-04-19T02:03:44.173296+0000","last_clean":"2024-04-19T02:03:44.173296+0000","last_became_active":"2024-04-19T02:03:11.677316+0000","last_became_peered":"2024-04-19T02:03:11.677316+0000","last_unstale":"2024-04-19T02:03:44.173296+0000","last_undegraded":"2024-04-19T02:03:44.173296+0000","last_fullsized":"2024-04-19T02:03:44.173296+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:19.091140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089705299999999998,"stat_sum":{"num_bytes":2834432,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":30,"num_read_kb":13423,"num_write":17,"num_write_kb":5072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8,6,4,10,1,7],"acting":[3,0,8,6,4,10,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"b","length":"1"},{"start":"e","length":"2"},{"start":"11","length":"1"}]},{"pgid":"3.a","version":"60'73","reported_seq":250,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-19T02:03:44.198821+0000","last_change":"2024-04-19T02:03:44.202176+0000","last_active":"2024-04-19T02:03:44.198821+0000","last_peered":"2024-04-19T02:03:44.198821+0000","last_clean":"2024-04-19T02:03:44.198821+0000","last_became_active":"2024-04-19T02:03:11.727996+0000","last_became_peered":"2024-04-19T02:03:11.727996+0000","last_unstale":"2024-04-19T02:03:44.198821+0000","last_undegraded":"2024-04-19T02:03:44.198821+0000","last_fullsized":"2024-04-19T02:03:44.198821+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:17.931487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022495370000000001,"stat_sum":{"num_bytes":11534336,"num_objects":7,"num_object_clones":3,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":16133,"num_write":85,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,1,9,5,7,3,2],"acting":[11,8,1,9,5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"b","length":"1"},{"start":"e","length":"2"},{"start":"11","length":"1"}]},{"pgid":"3.9","version":"60'27","reported_seq":205,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-19T02:03:44.196993+0000","last_change":"2024-04-19T02:03:44.199215+0000","last_active":"2024-04-19T02:03:44.196993+0000","last_peered":"2024-04-19T02:03:44.196993+0000","last_clean":"2024-04-19T02:03:44.196993+0000","last_became_active":"2024-04-19T02:03:11.729255+0000","last_became_peered":"2024-04-19T02:03:11.729255+0000","last_unstale":"2024-04-19T02:03:44.196993+0000","last_undegraded":"2024-04-19T02:03:44.196993+0000","last_fullsized":"2024-04-19T02:03:44.196993+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:31.434665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023072803999999999,"stat_sum":{"num_bytes":3833856,"num_objects":4,"num_object_clones":1,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":14,"num_read_kb":4615,"num_write":18,"num_write_kb":4480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,10,2,9,1,7,11],"acting":[4,3,10,2,9,1,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b","length":"1"},{"start":"e","length":"2"},{"start":"11","length":"1"}]},{"pgid":"3.8","version":"57'63","reported_seq":244,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-19T02:03:44.176488+0000","last_change":"2024-04-19T02:03:44.177436+0000","last_active":"2024-04-19T02:03:44.176488+0000","last_peered":"2024-04-19T02:03:44.176488+0000","last_clean":"2024-04-19T02:03:44.176488+0000","last_became_active":"2024-04-19T02:03:11.679384+0000","last_became_peered":"2024-04-19T02:03:11.679384+0000","last_unstale":"2024-04-19T02:03:44.176488+0000","last_undegraded":"2024-04-19T02:03:44.176488+0000","last_fullsized":"2024-04-19T02:03:44.176488+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:13.394718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041828330000000004,"stat_sum":{"num_bytes":9764864,"num_objects":6,"num_object_clones":3,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":26,"num_read_kb":10140,"num_write":32,"num_write_kb":9680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7,11,5,9,6,10],"acting":[4,1,7,11,5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b","length":"1"},{"start":"e","length":"2"},{"start":"11","length":"1"}]},{"pgid":"1.0","version":"21'137","reported_seq":265,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-19T02:03:44.166520+0000","last_change":"2024-04-19T02:02:59.974166+0000","last_active":"2024-04-19T02:03:44.166520+0000","last_peered":"2024-04-19T02:03:44.166520+0000","last_clean":"2024-04-19T02:03:44.166520+0000","last_became_active":"2024-04-19T02:02:59.972005+0000","last_became_peered":"2024-04-19T02:02:59.972005+0000","last_unstale":"2024-04-19T02:03:44.166520+0000","last_undegraded":"2024-04-19T02:03:44.166520+0000","last_fullsized":"2024-04-19T02:03:44.166520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:02:58.486225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:02:58.486225+0000","last_clean_scrub_stamp":"2024-04-19T02:02:58.486225+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:20.275707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":86,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-19T02:03:44.178671+0000","last_change":"2024-04-19T02:03:03.026806+0000","last_active":"2024-04-19T02:03:44.178671+0000","last_peered":"2024-04-19T02:03:44.178671+0000","last_clean":"2024-04-19T02:03:44.178671+0000","last_became_active":"2024-04-19T02:03:03.026063+0000","last_became_peered":"2024-04-19T02:03:03.026063+0000","last_unstale":"2024-04-19T02:03:44.178671+0000","last_undegraded":"2024-04-19T02:03:44.178671+0000","last_fullsized":"2024-04-19T02:03:44.178671+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:19.673016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"57'118","reported_seq":307,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-19T02:03:44.172780+0000","last_change":"2024-04-19T02:03:44.173190+0000","last_active":"2024-04-19T02:03:44.172780+0000","last_peered":"2024-04-19T02:03:44.172780+0000","last_clean":"2024-04-19T02:03:44.172780+0000","last_became_active":"2024-04-19T02:03:11.630475+0000","last_became_peered":"2024-04-19T02:03:11.630475+0000","last_unstale":"2024-04-19T02:03:44.172780+0000","last_undegraded":"2024-04-19T02:03:44.172780+0000","last_fullsized":"2024-04-19T02:03:44.172780+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:30.433161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061805900000000001,"stat_sum":{"num_bytes":10469376,"num_objects":9,"num_object_clones":2,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":51,"num_read_kb":14363,"num_write":93,"num_write_kb":16048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10,4,8,1,11,9],"acting":[3,5,10,4,8,1,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"b","length":"1"},{"start":"e","length":"2"},{"start":"11","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":86,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-19T02:03:44.166498+0000","last_change":"2024-04-19T02:03:02.561076+0000","last_active":"2024-04-19T02:03:44.166498+0000","last_peered":"2024-04-19T02:03:44.166498+0000","last_clean":"2024-04-19T02:03:44.166498+0000","last_became_active":"2024-04-19T02:03:02.560210+0000","last_became_peered":"2024-04-19T02:03:02.560210+0000","last_unstale":"2024-04-19T02:03:44.166498+0000","last_undegraded":"2024-04-19T02:03:44.166498+0000","last_fullsized":"2024-04-19T02:03:44.166498+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:16.900004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"47'21","reported_seq":178,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-19T02:03:40.214405+0000","last_change":"2024-04-19T02:03:39.055678+0000","last_active":"2024-04-19T02:03:40.214405+0000","last_peered":"2024-04-19T02:03:40.214405+0000","last_clean":"2024-04-19T02:03:40.214405+0000","last_became_active":"2024-04-19T02:03:11.633394+0000","last_became_peered":"2024-04-19T02:03:11.633394+0000","last_unstale":"2024-04-19T02:03:40.214405+0000","last_undegraded":"2024-04-19T02:03:40.214405+0000","last_fullsized":"2024-04-19T02:03:40.214405+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:38.385676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026699340000000001,"stat_sum":{"num_bytes":4096000,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":1350,"num_write":15,"num_write_kb":4528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,4,5,10,7,9,1],"acting":[0,11,4,5,10,7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"8","length":"1"},{"start":"a","length":"2"},{"start":"d","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":81,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-19T02:03:44.172335+0000","last_change":"2024-04-19T02:03:09.594947+0000","last_active":"2024-04-19T02:03:44.172335+0000","last_peered":"2024-04-19T02:03:44.172335+0000","last_clean":"2024-04-19T02:03:44.172335+0000","last_became_active":"2024-04-19T02:03:09.594072+0000","last_became_peered":"2024-04-19T02:03:09.594072+0000","last_unstale":"2024-04-19T02:03:44.172335+0000","last_undegraded":"2024-04-19T02:03:44.172335+0000","last_fullsized":"2024-04-19T02:03:44.172335+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:29.461650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":163,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-19T02:03:43.098976+0000","last_change":"2024-04-19T02:03:41.146891+0000","last_active":"2024-04-19T02:03:43.098976+0000","last_peered":"2024-04-19T02:03:43.098976+0000","last_clean":"2024-04-19T02:03:43.098976+0000","last_became_active":"2024-04-19T02:03:11.678078+0000","last_became_peered":"2024-04-19T02:03:11.678078+0000","last_unstale":"2024-04-19T02:03:43.098976+0000","last_undegraded":"2024-04-19T02:03:43.098976+0000","last_fullsized":"2024-04-19T02:03:43.098976+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:39.248619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016461620000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,9,1,6,4,0,10],"acting":[5,7,9,1,6,4,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"b","length":"1"},{"start":"e","length":"2"}]},{"pgid":"2.2","version":"0'0","reported_seq":86,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-19T02:03:44.166773+0000","last_change":"2024-04-19T02:03:02.560133+0000","last_active":"2024-04-19T02:03:44.166773+0000","last_peered":"2024-04-19T02:03:44.166773+0000","last_clean":"2024-04-19T02:03:44.166773+0000","last_became_active":"2024-04-19T02:03:02.559377+0000","last_became_peered":"2024-04-19T02:03:02.559377+0000","last_unstale":"2024-04-19T02:03:44.166773+0000","last_undegraded":"2024-04-19T02:03:44.166773+0000","last_fullsized":"2024-04-19T02:03:44.166773+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:30.728400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"57'24","reported_seq":191,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-19T02:03:43.129328+0000","last_change":"2024-04-19T02:03:41.181333+0000","last_active":"2024-04-19T02:03:43.129328+0000","last_peered":"2024-04-19T02:03:43.129328+0000","last_clean":"2024-04-19T02:03:43.129328+0000","last_became_active":"2024-04-19T02:03:11.677099+0000","last_became_peered":"2024-04-19T02:03:11.677099+0000","last_unstale":"2024-04-19T02:03:43.129328+0000","last_undegraded":"2024-04-19T02:03:43.129328+0000","last_fullsized":"2024-04-19T02:03:43.129328+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:36.589402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.028615848999999999,"stat_sum":{"num_bytes":4276224,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":2369,"num_write":53,"num_write_kb":4176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,0,3,4,7,9,6],"acting":[10,8,0,3,4,7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"b","length":"1"},{"start":"e","length":"2"}]},{"pgid":"3.5","version":"45'33","reported_seq":198,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-19T02:03:43.129475+0000","last_change":"2024-04-19T02:03:41.152713+0000","last_active":"2024-04-19T02:03:43.129475+0000","last_peered":"2024-04-19T02:03:43.129475+0000","last_clean":"2024-04-19T02:03:43.129475+0000","last_became_active":"2024-04-19T02:03:11.728949+0000","last_became_peered":"2024-04-19T02:03:11.728949+0000","last_unstale":"2024-04-19T02:03:43.129475+0000","last_undegraded":"2024-04-19T02:03:43.129475+0000","last_fullsized":"2024-04-19T02:03:43.129475+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:14.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015071609999999999,"stat_sum":{"num_bytes":2293760,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":36,"num_write_kb":3584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2,9,8,1,6,5],"acting":[10,3,2,9,8,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"b","length":"1"},{"start":"e","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":84,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-19T02:03:43.114799+0000","last_change":"2024-04-19T02:03:02.597582+0000","last_active":"2024-04-19T02:03:43.114799+0000","last_peered":"2024-04-19T02:03:43.114799+0000","last_clean":"2024-04-19T02:03:43.114799+0000","last_became_active":"2024-04-19T02:03:02.597118+0000","last_became_peered":"2024-04-19T02:03:02.597118+0000","last_unstale":"2024-04-19T02:03:43.114799+0000","last_undegraded":"2024-04-19T02:03:43.114799+0000","last_fullsized":"2024-04-19T02:03:43.114799+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:13.683143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"59'61","reported_seq":231,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-19T02:03:43.180531+0000","last_change":"2024-04-19T02:03:41.156783+0000","last_active":"2024-04-19T02:03:43.180531+0000","last_peered":"2024-04-19T02:03:43.180531+0000","last_clean":"2024-04-19T02:03:43.180531+0000","last_became_active":"2024-04-19T02:03:11.730165+0000","last_became_peered":"2024-04-19T02:03:11.730165+0000","last_unstale":"2024-04-19T02:03:43.180531+0000","last_undegraded":"2024-04-19T02:03:43.180531+0000","last_fullsized":"2024-04-19T02:03:43.180531+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:30.219139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015931614,"stat_sum":{"num_bytes":7651328,"num_objects":8,"num_object_clones":3,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":25,"num_read_kb":6443,"num_write":51,"num_write_kb":9296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,10,8,7,2,11,0],"acting":[1,5,10,8,7,2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"b","length":"1"},{"start":"e","length":"2"}]},{"pgid":"2.5","version":"0'0","reported_seq":86,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-19T02:03:44.166027+0000","last_change":"2024-04-19T02:03:02.595929+0000","last_active":"2024-04-19T02:03:44.166027+0000","last_peered":"2024-04-19T02:03:44.166027+0000","last_clean":"2024-04-19T02:03:44.166027+0000","last_became_active":"2024-04-19T02:03:02.595232+0000","last_became_peered":"2024-04-19T02:03:02.595232+0000","last_unstale":"2024-04-19T02:03:44.166027+0000","last_undegraded":"2024-04-19T02:03:44.166027+0000","last_fullsized":"2024-04-19T02:03:44.166027+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:04.077370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"57'20","reported_seq":190,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-19T02:03:44.173027+0000","last_change":"2024-04-19T02:03:44.173370+0000","last_active":"2024-04-19T02:03:44.173027+0000","last_peered":"2024-04-19T02:03:44.173027+0000","last_clean":"2024-04-19T02:03:44.173027+0000","last_became_active":"2024-04-19T02:03:11.675056+0000","last_became_peered":"2024-04-19T02:03:11.675056+0000","last_unstale":"2024-04-19T02:03:44.173027+0000","last_undegraded":"2024-04-19T02:03:44.173027+0000","last_fullsized":"2024-04-19T02:03:44.173027+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:21.774339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063423899999999998,"stat_sum":{"num_bytes":2162688,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,9,0,7,5,11,1],"acting":[3,6,9,0,7,5,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"b","length":"1"},{"start":"e","length":"2"},{"start":"11","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":84,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-19T02:03:43.113886+0000","last_change":"2024-04-19T02:03:02.988302+0000","last_active":"2024-04-19T02:03:43.113886+0000","last_peered":"2024-04-19T02:03:43.113886+0000","last_clean":"2024-04-19T02:03:43.113886+0000","last_became_active":"2024-04-19T02:03:02.988073+0000","last_became_peered":"2024-04-19T02:03:02.988073+0000","last_unstale":"2024-04-19T02:03:43.113886+0000","last_undegraded":"2024-04-19T02:03:43.113886+0000","last_fullsized":"2024-04-19T02:03:43.113886+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:28.455271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"56'4","reported_seq":157,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-19T02:03:40.215219+0000","last_change":"2024-04-19T02:03:39.055697+0000","last_active":"2024-04-19T02:03:40.215219+0000","last_peered":"2024-04-19T02:03:40.215219+0000","last_clean":"2024-04-19T02:03:40.215219+0000","last_became_active":"2024-04-19T02:03:11.673940+0000","last_became_peered":"2024-04-19T02:03:11.673940+0000","last_unstale":"2024-04-19T02:03:40.215219+0000","last_undegraded":"2024-04-19T02:03:40.215219+0000","last_fullsized":"2024-04-19T02:03:40.215219+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:14.043287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00244319,"stat_sum":{"num_bytes":180224,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":6,"num_read_kb":530,"num_write":3,"num_write_kb":176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9,8,4,5,6,10],"acting":[0,1,9,8,4,5,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"8","length":"1"},{"start":"a","length":"2"},{"start":"d","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":86,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-19T02:03:44.167001+0000","last_change":"2024-04-19T02:03:02.570839+0000","last_active":"2024-04-19T02:03:44.167001+0000","last_peered":"2024-04-19T02:03:44.167001+0000","last_clean":"2024-04-19T02:03:44.167001+0000","last_became_active":"2024-04-19T02:03:02.570116+0000","last_became_peered":"2024-04-19T02:03:02.570116+0000","last_unstale":"2024-04-19T02:03:44.167001+0000","last_undegraded":"2024-04-19T02:03:44.167001+0000","last_fullsized":"2024-04-19T02:03:44.167001+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:23.134425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":84885504,"num_objects":68,"num_object_clones":23,"num_object_copies":544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":6,"num_read":320,"num_read_kb":107813,"num_write":527,"num_write_kb":102144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":166404096,"data_stored":202178560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":676,"ondisk_log_size":676,"up":128,"acting":128,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":19,"seq":81604378636,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40460,"kb_used_data":9276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331380,"statfs":{"total":96636764160,"available":96595333120,"internally_reserved":0,"allocated":9498624,"data_stored":10570486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":19,"seq":81604378635,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41860,"kb_used_data":10672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329980,"statfs":{"total":96636764160,"available":96593899520,"internally_reserved":0,"allocated":10928128,"data_stored":12315607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411340,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50212,"kb_used_data":19032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321628,"statfs":{"total":96636764160,"available":96585347072,"internally_reserved":0,"allocated":19488768,"data_stored":23428874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411341,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48012,"kb_used_data":16824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323828,"statfs":{"total":96636764160,"available":96587599872,"internally_reserved":0,"allocated":17227776,"data_stored":20992035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411341,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41704,"kb_used_data":14684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330136,"statfs":{"total":96636764160,"available":96594059264,"internally_reserved":0,"allocated":15036416,"data_stored":17953347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411341,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45240,"kb_used_data":18212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94326600,"statfs":{"total":96636764160,"available":96590438400,"internally_reserved":0,"allocated":18649088,"data_stored":23191306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":18,"seq":77309411339,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32928,"kb_used_data":5900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338912,"statfs":{"total":96636764160,"available":96603045888,"internally_reserved":0,"allocated":6041600,"data_stored":7372554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411340,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42628,"kb_used_data":15600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329212,"statfs":{"total":96636764160,"available":96593113088,"internally_reserved":0,"allocated":15974400,"data_stored":17862410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":18,"seq":77309411340,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43104,"kb_used_data":16076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94328736,"statfs":{"total":96636764160,"available":96592625664,"internally_reserved":0,"allocated":16461824,"data_stored":20250378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":17,"seq":73014444044,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38260,"kb_used_data":11240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94333580,"statfs":{"total":96636764160,"available":96597585920,"internally_reserved":0,"allocated":11509760,"data_stored":12492016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":17,"seq":73014444044,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50404,"kb_used_data":19220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321436,"statfs":{"total":96636764160,"available":96585150464,"internally_reserved":0,"allocated":19681280,"data_stored":23617834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":17,"seq":73014444044,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44260,"kb_used_data":13076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327580,"statfs":{"total":96636764160,"available":96591441920,"internally_reserved":0,"allocated":13389824,"data_stored":15133936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10301440,"data_stored":11952128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13705216,"data_stored":16822272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8884224,"data_stored":10215424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":15798272,"data_stored":19861504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":15310848,"data_stored":17473536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":18485248,"data_stored":22700032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5378048,"data_stored":6983680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":17985536,"data_stored":22802432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":13828096,"data_stored":17027072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16551936,"data_stored":20594688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":11350016,"data_stored":12705792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":18825216,"data_stored":23040000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T02:03:45.499 INFO:tasks.ceph.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-04-19T02:03:45.535 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:45.537+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:45.636 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:45.637+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:45.659 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:45.659 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-04-19T02:03:45.659 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-04-19T02:03:45.659 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-04-19T02:03:45.660 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.660 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:237 3.2s0 3:407ecba7:::smithi05930959-22:2 [read 0~360448,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-19T02:03:45.660 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.646438+0000", 2024-04-19T02:03:45.660 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 29.013196161, 2024-04-19T02:03:45.660 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.024327603, 2024-04-19T02:03:45.660 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.660 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.660 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:45.660 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:45.660 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:45.660 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:45.660 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 237 2024-04-19T02:03:45.660 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.661 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.661 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.661 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.661 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646438+0000", 2024-04-19T02:03:45.661 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.661 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.661 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.661 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.661 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646438+0000", 2024-04-19T02:03:45.661 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.661 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.661 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.661 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.662 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646440+0000", 2024-04-19T02:03:45.662 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.254e-06 2024-04-19T02:03:45.662 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.662 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.662 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.662 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646447+0000", 2024-04-19T02:03:45.662 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.3289999999999996e-06 2024-04-19T02:03:45.662 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.662 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.662 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.662 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646463+0000", 2024-04-19T02:03:45.662 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5846e-05 2024-04-19T02:03:45.662 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.663 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.663 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.663 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646472+0000", 2024-04-19T02:03:45.663 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.2939999999999994e-06 2024-04-19T02:03:45.663 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.663 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.663 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.663 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646953+0000", 2024-04-19T02:03:45.663 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000480901 2024-04-19T02:03:45.663 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.663 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.663 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:03:45.663 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646971+0000", 2024-04-19T02:03:45.663 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8167000000000001e-05 2024-04-19T02:03:45.663 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.664 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.664 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.664 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.656667+0000", 2024-04-19T02:03:45.664 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0096952449999999999 2024-04-19T02:03:45.664 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.664 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.664 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:45.664 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.656833+0000", 2024-04-19T02:03:45.664 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00016697500000000001 2024-04-19T02:03:45.664 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.664 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.664 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.664 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.670766+0000", 2024-04-19T02:03:45.664 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.013932592000000001 2024-04-19T02:03:45.665 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.665 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.665 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.665 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.665 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.665 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:238 3.2s0 3:407ecba7:::smithi05930959-22:2 [read 0~360448,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-19T02:03:45.665 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.646510+0000", 2024-04-19T02:03:45.665 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 29.013124878999999, 2024-04-19T02:03:45.665 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.028744658999999999, 2024-04-19T02:03:45.665 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.665 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.665 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:45.665 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:45.665 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:45.665 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:45.666 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 238 2024-04-19T02:03:45.666 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.666 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.666 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.666 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.666 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646510+0000", 2024-04-19T02:03:45.666 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.666 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.666 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.666 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.666 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646510+0000", 2024-04-19T02:03:45.666 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.667 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.667 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.667 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.667 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646511+0000", 2024-04-19T02:03:45.667 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.8599999999999997e-07 2024-04-19T02:03:45.667 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.667 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.667 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.667 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646517+0000", 2024-04-19T02:03:45.667 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.1700000000000002e-06 2024-04-19T02:03:45.667 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.667 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.667 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.667 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646528+0000", 2024-04-19T02:03:45.667 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1202999999999999e-05 2024-04-19T02:03:45.668 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.668 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.668 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.668 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646537+0000", 2024-04-19T02:03:45.668 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7460000000000006e-06 2024-04-19T02:03:45.668 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.668 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.668 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.668 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.647105+0000", 2024-04-19T02:03:45.668 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00056801599999999996 2024-04-19T02:03:45.668 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.668 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.668 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:03:45.668 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.647127+0000", 2024-04-19T02:03:45.668 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.175e-05 2024-04-19T02:03:45.669 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.669 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.669 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.669 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.657504+0000", 2024-04-19T02:03:45.669 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010377961 2024-04-19T02:03:45.669 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.669 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.669 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:45.669 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.657826+0000", 2024-04-19T02:03:45.669 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00032128300000000001 2024-04-19T02:03:45.669 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.669 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.669 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.670 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.675254+0000", 2024-04-19T02:03:45.670 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.017428644 2024-04-19T02:03:45.670 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.670 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.670 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.670 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.670 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.670 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:239 3.2s0 3:407ecba7:::smithi05930959-22:2 [read 0~360448,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-19T02:03:45.670 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.646549+0000", 2024-04-19T02:03:45.670 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 29.013085407999998, 2024-04-19T02:03:45.670 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.029381570999999999, 2024-04-19T02:03:45.670 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.670 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.670 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:45.671 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:45.671 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:45.671 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:45.671 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 239 2024-04-19T02:03:45.671 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.671 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.671 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.671 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.671 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646549+0000", 2024-04-19T02:03:45.671 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.671 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.671 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.671 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.671 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646549+0000", 2024-04-19T02:03:45.671 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.672 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.672 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.672 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.672 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646550+0000", 2024-04-19T02:03:45.672 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8400000000000004e-07 2024-04-19T02:03:45.672 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.672 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.672 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.672 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646555+0000", 2024-04-19T02:03:45.672 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9989999999999999e-06 2024-04-19T02:03:45.672 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.672 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.672 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.672 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646567+0000", 2024-04-19T02:03:45.672 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.202e-05 2024-04-19T02:03:45.673 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.673 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.673 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.673 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646575+0000", 2024-04-19T02:03:45.673 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.7640000000000005e-06 2024-04-19T02:03:45.673 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.673 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.673 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.673 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.647229+0000", 2024-04-19T02:03:45.673 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00065415299999999998 2024-04-19T02:03:45.673 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.673 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.673 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:03:45.673 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.647255+0000", 2024-04-19T02:03:45.674 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5795999999999998e-05 2024-04-19T02:03:45.674 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.674 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.674 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.674 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.659835+0000", 2024-04-19T02:03:45.674 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012580713 2024-04-19T02:03:45.674 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.674 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.674 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:45.674 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.660090+0000", 2024-04-19T02:03:45.674 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00025430899999999998 2024-04-19T02:03:45.674 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.674 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.674 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.675 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.675931+0000", 2024-04-19T02:03:45.675 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.015841133 2024-04-19T02:03:45.675 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.675 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.675 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.675 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.675 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.675 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:349 3.2s0 3:43b7722e:::smithi05930959-10:head [copy-from ver 11] snapc 8=[8,6,5] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:45.675 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.746333+0000", 2024-04-19T02:03:45.675 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 20.913301251, 2024-04-19T02:03:45.675 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.034219922999999999, 2024-04-19T02:03:45.675 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.675 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.675 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:45.675 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:45.676 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:45.676 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:45.676 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 349 2024-04-19T02:03:45.676 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.676 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.676 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.676 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.676 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746333+0000", 2024-04-19T02:03:45.676 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.676 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.676 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.676 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.676 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746333+0000", 2024-04-19T02:03:45.676 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.677 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.677 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.677 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.677 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746337+0000", 2024-04-19T02:03:45.677 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4769999999999999e-06 2024-04-19T02:03:45.677 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.677 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.677 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.677 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746352+0000", 2024-04-19T02:03:45.677 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5277e-05 2024-04-19T02:03:45.677 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.677 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.677 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.677 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746387+0000", 2024-04-19T02:03:45.677 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4436e-05 2024-04-19T02:03:45.678 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.678 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.678 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.678 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746403+0000", 2024-04-19T02:03:45.678 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6475999999999999e-05 2024-04-19T02:03:45.678 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.678 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.678 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.678 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746490+0000", 2024-04-19T02:03:45.678 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7213000000000007e-05 2024-04-19T02:03:45.678 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.678 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.678 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "op must wait for map", 2024-04-19T02:03:45.678 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746497+0000", 2024-04-19T02:03:45.679 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.0520000000000004e-06 2024-04-19T02:03:45.679 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.679 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.679 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.679 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749865+0000", 2024-04-19T02:03:45.679 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0033676069999999999 2024-04-19T02:03:45.679 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.679 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.679 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:45.679 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749938+0000", 2024-04-19T02:03:45.679 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.3530999999999998e-05 2024-04-19T02:03:45.679 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.679 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.679 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:45.679 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.768855+0000", 2024-04-19T02:03:45.680 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.018916493 2024-04-19T02:03:45.680 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.680 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.680 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:45.680 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.771377+0000", 2024-04-19T02:03:45.680 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0025221520000000002 2024-04-19T02:03:45.680 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.680 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.680 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:45.680 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.780460+0000", 2024-04-19T02:03:45.680 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0090830389999999993 2024-04-19T02:03:45.680 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.680 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.680 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.680 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.780553+0000", 2024-04-19T02:03:45.681 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3170000000000004e-05 2024-04-19T02:03:45.681 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.681 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.681 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.681 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.681 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.681 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:350 3.2s0 3:43b7722e:::smithi05930959-10:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:45.681 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.746450+0000", 2024-04-19T02:03:45.681 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 20.913184592, 2024-04-19T02:03:45.681 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.024575331999999998, 2024-04-19T02:03:45.681 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.681 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.681 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:45.681 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:45.681 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:45.682 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:45.682 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 350 2024-04-19T02:03:45.682 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.682 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.682 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.682 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.682 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746450+0000", 2024-04-19T02:03:45.682 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.682 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.682 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.682 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.682 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746450+0000", 2024-04-19T02:03:45.682 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.682 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.682 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.683 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.683 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746451+0000", 2024-04-19T02:03:45.683 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0400000000000005e-07 2024-04-19T02:03:45.683 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.683 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.683 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.683 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746456+0000", 2024-04-19T02:03:45.683 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5199999999999997e-06 2024-04-19T02:03:45.683 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.683 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.683 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.683 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746470+0000", 2024-04-19T02:03:45.683 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3935e-05 2024-04-19T02:03:45.683 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.684 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.684 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.684 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746481+0000", 2024-04-19T02:03:45.684 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0919e-05 2024-04-19T02:03:45.684 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.684 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.684 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.684 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746552+0000", 2024-04-19T02:03:45.684 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.0235999999999996e-05 2024-04-19T02:03:45.684 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.684 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.684 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:03:45.684 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746557+0000", 2024-04-19T02:03:45.684 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.6570000000000002e-06 2024-04-19T02:03:45.684 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.685 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.685 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.685 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750143+0000", 2024-04-19T02:03:45.685 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0035860549999999999 2024-04-19T02:03:45.685 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.685 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.685 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for blocked object", 2024-04-19T02:03:45.685 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750177+0000", 2024-04-19T02:03:45.685 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3796e-05 2024-04-19T02:03:45.685 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.685 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.685 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.685 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.770842+0000", 2024-04-19T02:03:45.685 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.020664657999999999 2024-04-19T02:03:45.685 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.686 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.686 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:45.686 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.770923+0000", 2024-04-19T02:03:45.686 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0869000000000001e-05 2024-04-19T02:03:45.686 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.686 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.686 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.686 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.771025+0000", 2024-04-19T02:03:45.686 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010278299999999999 2024-04-19T02:03:45.686 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.686 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.686 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.686 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.686 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.687 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:422 3.2s0 3:43b7722e:::smithi05930959-10:head [copy-from ver 11] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:45.687 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.193952+0000", 2024-04-19T02:03:45.687 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 15.465682253000001, 2024-04-19T02:03:45.687 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.044114190999999997, 2024-04-19T02:03:45.687 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.687 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.687 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:45.687 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:45.687 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:45.687 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:45.687 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 422 2024-04-19T02:03:45.687 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.687 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.687 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.688 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.688 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.193952+0000", 2024-04-19T02:03:45.688 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.688 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.688 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.688 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.688 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.193952+0000", 2024-04-19T02:03:45.688 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.688 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.688 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.688 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.688 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.193955+0000", 2024-04-19T02:03:45.688 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3159999999999999e-06 2024-04-19T02:03:45.688 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.688 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.689 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.689 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.193968+0000", 2024-04-19T02:03:45.689 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3726999999999999e-05 2024-04-19T02:03:45.689 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.689 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.689 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.689 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.193993+0000", 2024-04-19T02:03:45.689 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4499e-05 2024-04-19T02:03:45.689 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.689 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.689 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.689 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194005+0000", 2024-04-19T02:03:45.689 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1770000000000001e-05 2024-04-19T02:03:45.690 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.690 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.690 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.690 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194109+0000", 2024-04-19T02:03:45.690 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000103904 2024-04-19T02:03:45.690 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.690 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.690 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:45.690 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194249+0000", 2024-04-19T02:03:45.690 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014036699999999999 2024-04-19T02:03:45.690 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.690 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.690 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:45.690 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.224308+0000", 2024-04-19T02:03:45.690 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.030059210999999999 2024-04-19T02:03:45.691 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.691 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.691 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:45.691 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.237172+0000", 2024-04-19T02:03:45.691 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012863734999999999 2024-04-19T02:03:45.691 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.691 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.691 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:45.691 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.237230+0000", 2024-04-19T02:03:45.691 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.8238e-05 2024-04-19T02:03:45.691 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.691 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.691 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.691 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.238067+0000", 2024-04-19T02:03:45.692 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000836424 2024-04-19T02:03:45.692 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.692 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.692 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.692 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.692 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.692 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:423 3.2s0 3:43b7722e:::smithi05930959-10:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:45.692 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194040+0000", 2024-04-19T02:03:45.692 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 15.465594922999999, 2024-04-19T02:03:45.692 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.040466800999999997, 2024-04-19T02:03:45.692 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.692 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.692 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:45.692 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:45.692 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:45.692 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:45.693 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 423 2024-04-19T02:03:45.693 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.693 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.693 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.693 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.693 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194040+0000", 2024-04-19T02:03:45.693 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.693 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.693 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.693 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.693 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194040+0000", 2024-04-19T02:03:45.693 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.693 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.693 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.694 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.694 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194040+0000", 2024-04-19T02:03:45.694 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.4700000000000001e-07 2024-04-19T02:03:45.694 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.694 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.694 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.694 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194044+0000", 2024-04-19T02:03:45.694 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9049999999999999e-06 2024-04-19T02:03:45.694 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.694 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.694 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.694 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194056+0000", 2024-04-19T02:03:45.694 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1567e-05 2024-04-19T02:03:45.694 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.695 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.695 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.695 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194063+0000", 2024-04-19T02:03:45.695 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.1539999999999996e-06 2024-04-19T02:03:45.695 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.695 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.695 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.695 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194511+0000", 2024-04-19T02:03:45.695 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00044832999999999999 2024-04-19T02:03:45.695 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.695 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.695 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for blocked object", 2024-04-19T02:03:45.695 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194561+0000", 2024-04-19T02:03:45.695 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9336999999999999e-05 2024-04-19T02:03:45.696 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.696 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.696 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.696 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.234276+0000", 2024-04-19T02:03:45.696 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.039715156000000001 2024-04-19T02:03:45.696 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.696 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.696 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:45.696 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.234405+0000", 2024-04-19T02:03:45.696 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00012947100000000001 2024-04-19T02:03:45.696 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.696 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.696 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.696 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.234507+0000", 2024-04-19T02:03:45.697 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000101234 2024-04-19T02:03:45.697 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.697 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.697 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.697 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.697 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.697 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:424 3.2s0 3:407ecba7:::smithi05930959-22:head [append 0~0] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:45.697 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194090+0000", 2024-04-19T02:03:45.697 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 15.465544178, 2024-04-19T02:03:45.697 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.04103888, 2024-04-19T02:03:45.697 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.697 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.697 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:45.697 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:45.697 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:45.698 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:45.698 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 424 2024-04-19T02:03:45.698 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.698 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.698 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.698 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.698 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194090+0000", 2024-04-19T02:03:45.698 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.698 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.698 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.698 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.698 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194090+0000", 2024-04-19T02:03:45.698 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.698 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.699 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.699 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.699 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194091+0000", 2024-04-19T02:03:45.699 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.9599999999999999e-07 2024-04-19T02:03:45.699 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.699 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.699 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.699 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194095+0000", 2024-04-19T02:03:45.699 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.019e-06 2024-04-19T02:03:45.699 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.699 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.699 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.699 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194104+0000", 2024-04-19T02:03:45.699 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.6030000000000005e-06 2024-04-19T02:03:45.699 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.699 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.700 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.700 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194110+0000", 2024-04-19T02:03:45.700 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.6880000000000002e-06 2024-04-19T02:03:45.700 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.700 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.700 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.700 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194597+0000", 2024-04-19T02:03:45.700 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00048649499999999998 2024-04-19T02:03:45.700 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.700 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.700 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:45.700 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194669+0000", 2024-04-19T02:03:45.700 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.2081000000000003e-05 2024-04-19T02:03:45.701 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.701 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.701 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:45.701 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195298+0000", 2024-04-19T02:03:45.701 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000628635 2024-04-19T02:03:45.701 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.701 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.701 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:45.701 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.201226+0000", 2024-04-19T02:03:45.701 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0059289199999999998 2024-04-19T02:03:45.701 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.701 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.701 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:45.701 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235058+0000", 2024-04-19T02:03:45.701 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.033831539000000001 2024-04-19T02:03:45.702 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.702 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.702 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.702 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235129+0000", 2024-04-19T02:03:45.702 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.1303999999999994e-05 2024-04-19T02:03:45.702 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.702 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.702 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.702 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.702 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.702 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:425 3.2s0 3:407ecba7:::smithi05930959-22:head [rollback a] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:45.702 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194118+0000", 2024-04-19T02:03:45.702 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 15.465516715, 2024-04-19T02:03:45.702 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.041424615999999997, 2024-04-19T02:03:45.703 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.703 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.703 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:45.703 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:45.703 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:45.703 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:45.703 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 425 2024-04-19T02:03:45.703 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.703 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.703 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.703 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.703 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194118+0000", 2024-04-19T02:03:45.703 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.703 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.704 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.704 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.704 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194118+0000", 2024-04-19T02:03:45.704 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.704 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.704 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.704 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.704 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194118+0000", 2024-04-19T02:03:45.704 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7799999999999997e-07 2024-04-19T02:03:45.704 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.704 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.704 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.704 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194120+0000", 2024-04-19T02:03:45.705 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9740000000000001e-06 2024-04-19T02:03:45.705 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.705 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.705 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.705 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194124+0000", 2024-04-19T02:03:45.705 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1540000000000004e-06 2024-04-19T02:03:45.705 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.705 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.705 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.705 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194127+0000", 2024-04-19T02:03:45.705 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1379999999999999e-06 2024-04-19T02:03:45.705 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.705 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.705 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.706 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195791+0000", 2024-04-19T02:03:45.706 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0016630779999999999 2024-04-19T02:03:45.706 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.706 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.706 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:45.706 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195884+0000", 2024-04-19T02:03:45.706 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3885e-05 2024-04-19T02:03:45.706 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.706 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.706 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:45.706 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.196827+0000", 2024-04-19T02:03:45.706 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00094285400000000002 2024-04-19T02:03:45.706 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.706 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.706 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:45.707 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.201229+0000", 2024-04-19T02:03:45.707 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0044017359999999998 2024-04-19T02:03:45.707 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.707 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.707 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:45.707 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235479+0000", 2024-04-19T02:03:45.707 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.034249891999999997 2024-04-19T02:03:45.707 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.707 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.707 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.707 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235543+0000", 2024-04-19T02:03:45.707 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.3627000000000006e-05 2024-04-19T02:03:45.707 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.707 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.708 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.708 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.708 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.708 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:426 3.2s0 3:407ecba7:::smithi05930959-22:head [append 0~0] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:45.708 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194132+0000", 2024-04-19T02:03:45.708 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 15.465503104, 2024-04-19T02:03:45.708 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.041767809000000003, 2024-04-19T02:03:45.708 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.708 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.708 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:45.708 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:45.708 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:45.708 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:45.709 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 426 2024-04-19T02:03:45.709 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.709 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.709 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.709 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.709 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194132+0000", 2024-04-19T02:03:45.709 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.709 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.709 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.709 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.709 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194132+0000", 2024-04-19T02:03:45.709 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.709 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.709 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.710 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.710 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194132+0000", 2024-04-19T02:03:45.710 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5199999999999998e-07 2024-04-19T02:03:45.710 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.710 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.710 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.710 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194133+0000", 2024-04-19T02:03:45.710 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.624e-06 2024-04-19T02:03:45.710 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.710 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.710 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.710 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194137+0000", 2024-04-19T02:03:45.710 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3550000000000001e-06 2024-04-19T02:03:45.710 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.711 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.711 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.711 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194140+0000", 2024-04-19T02:03:45.711 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2229999999999999e-06 2024-04-19T02:03:45.711 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.711 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.711 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.711 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.198324+0000", 2024-04-19T02:03:45.711 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0041834799999999998 2024-04-19T02:03:45.711 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.711 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.711 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:45.711 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.198430+0000", 2024-04-19T02:03:45.711 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010597100000000001 2024-04-19T02:03:45.711 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.712 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.712 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:45.712 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.199339+0000", 2024-04-19T02:03:45.712 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00090987100000000003 2024-04-19T02:03:45.712 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.712 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.712 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:45.712 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.201230+0000", 2024-04-19T02:03:45.712 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0018903170000000001 2024-04-19T02:03:45.712 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.712 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.712 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:45.712 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235836+0000", 2024-04-19T02:03:45.712 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.034606382999999998 2024-04-19T02:03:45.713 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.713 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.713 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.713 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235899+0000", 2024-04-19T02:03:45.713 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.3232999999999996e-05 2024-04-19T02:03:45.713 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.713 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.713 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.713 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.713 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.713 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:437 3.bs0 3:d6e3a590:::smithi05930959-43:head [copy-from ver 51] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:45.713 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194173+0000", 2024-04-19T02:03:45.713 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 15.465461857999999, 2024-04-19T02:03:45.713 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.054675901999999998, 2024-04-19T02:03:45.713 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.714 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.714 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:45.714 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:45.714 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:45.714 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:45.714 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 437 2024-04-19T02:03:45.714 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.714 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.714 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.714 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.714 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194173+0000", 2024-04-19T02:03:45.714 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.714 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.715 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.715 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.715 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194173+0000", 2024-04-19T02:03:45.715 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.715 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.715 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.715 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.715 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194173+0000", 2024-04-19T02:03:45.715 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1100000000000002e-07 2024-04-19T02:03:45.715 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.715 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.715 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.715 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194179+0000", 2024-04-19T02:03:45.715 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.122e-06 2024-04-19T02:03:45.715 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.716 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.716 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.716 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194185+0000", 2024-04-19T02:03:45.716 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.3109999999999996e-06 2024-04-19T02:03:45.716 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.716 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.716 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.716 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194188+0000", 2024-04-19T02:03:45.716 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0230000000000001e-06 2024-04-19T02:03:45.716 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.716 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.716 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.716 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194246+0000", 2024-04-19T02:03:45.716 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.8865e-05 2024-04-19T02:03:45.717 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.717 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.717 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:45.717 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194356+0000", 2024-04-19T02:03:45.717 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000109558 2024-04-19T02:03:45.717 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.717 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.717 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:45.717 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.240077+0000", 2024-04-19T02:03:45.717 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.045721108000000003 2024-04-19T02:03:45.717 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.717 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.717 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:45.717 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.244667+0000", 2024-04-19T02:03:45.717 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0045899000000000001 2024-04-19T02:03:45.718 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.718 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.718 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:45.718 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.248766+0000", 2024-04-19T02:03:45.718 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0040986579999999998 2024-04-19T02:03:45.718 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.718 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.718 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.718 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.248849+0000", 2024-04-19T02:03:45.718 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3046000000000004e-05 2024-04-19T02:03:45.718 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.718 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.718 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.718 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.719 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.719 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:438 3.bs0 3:d6e3a590:::smithi05930959-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:45.719 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194196+0000", 2024-04-19T02:03:45.719 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 15.465439045, 2024-04-19T02:03:45.719 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.047992689999999998, 2024-04-19T02:03:45.719 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.719 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.719 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:45.719 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:45.719 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:45.719 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:45.719 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 438 2024-04-19T02:03:45.719 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.719 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.719 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.720 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.720 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194196+0000", 2024-04-19T02:03:45.720 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.720 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.720 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.720 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.720 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194196+0000", 2024-04-19T02:03:45.720 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.720 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.720 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.720 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.720 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194196+0000", 2024-04-19T02:03:45.720 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4700000000000002e-07 2024-04-19T02:03:45.720 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.720 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.721 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.721 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194198+0000", 2024-04-19T02:03:45.721 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4320000000000002e-06 2024-04-19T02:03:45.721 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.721 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.721 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.721 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194204+0000", 2024-04-19T02:03:45.721 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0850000000000002e-06 2024-04-19T02:03:45.721 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.721 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.721 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.721 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194209+0000", 2024-04-19T02:03:45.721 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5090000000000004e-06 2024-04-19T02:03:45.721 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.722 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.722 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.722 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194533+0000", 2024-04-19T02:03:45.722 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00032412499999999998 2024-04-19T02:03:45.722 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.722 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.722 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for blocked object", 2024-04-19T02:03:45.722 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194593+0000", 2024-04-19T02:03:45.722 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.9854e-05 2024-04-19T02:03:45.722 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.722 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.722 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.722 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.242048+0000", 2024-04-19T02:03:45.722 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.047455152 2024-04-19T02:03:45.723 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.723 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.723 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:45.723 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.242113+0000", 2024-04-19T02:03:45.723 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.4573000000000007e-05 2024-04-19T02:03:45.723 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.723 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.723 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.723 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.242188+0000", 2024-04-19T02:03:45.723 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.5612999999999996e-05 2024-04-19T02:03:45.723 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.723 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.723 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.723 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.724 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.724 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:439 3.2s0 3:43ce42f3:::smithi05930959-27:head [append 0~0] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:45.724 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194231+0000", 2024-04-19T02:03:45.724 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 15.465403611999999, 2024-04-19T02:03:45.724 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.041997487, 2024-04-19T02:03:45.724 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.724 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.724 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:45.724 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:45.724 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:45.724 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:45.724 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 439 2024-04-19T02:03:45.724 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.724 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.724 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.724 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.725 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194231+0000", 2024-04-19T02:03:45.725 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.725 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.725 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.725 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.725 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194231+0000", 2024-04-19T02:03:45.725 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.725 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.725 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.725 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.725 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194232+0000", 2024-04-19T02:03:45.725 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7999999999999995e-07 2024-04-19T02:03:45.725 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.726 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.726 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.726 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194236+0000", 2024-04-19T02:03:45.726 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2139999999999998e-06 2024-04-19T02:03:45.726 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.726 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.726 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.726 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194243+0000", 2024-04-19T02:03:45.726 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.5979999999999999e-06 2024-04-19T02:03:45.726 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.726 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.726 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.726 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194248+0000", 2024-04-19T02:03:45.726 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.3370000000000001e-06 2024-04-19T02:03:45.727 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.727 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.727 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.727 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.204081+0000", 2024-04-19T02:03:45.727 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0098332009999999997 2024-04-19T02:03:45.727 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.727 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.727 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:45.727 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.204143+0000", 2024-04-19T02:03:45.727 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.156e-05 2024-04-19T02:03:45.727 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.727 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.727 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:45.727 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.204875+0000", 2024-04-19T02:03:45.728 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00073241099999999998 2024-04-19T02:03:45.728 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.728 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.728 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:45.728 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.208933+0000", 2024-04-19T02:03:45.728 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0040575990000000003 2024-04-19T02:03:45.728 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.728 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.728 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:45.728 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.236159+0000", 2024-04-19T02:03:45.728 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.027226274000000002 2024-04-19T02:03:45.728 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.728 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.728 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.729 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.236229+0000", 2024-04-19T02:03:45.729 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9713000000000002e-05 2024-04-19T02:03:45.729 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.729 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.729 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.729 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.729 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.729 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:440 3.2s0 3:43ce42f3:::smithi05930959-27:head [rollback a] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:45.729 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194253+0000", 2024-04-19T02:03:45.729 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 15.465381372, 2024-04-19T02:03:45.729 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.042233729999999997, 2024-04-19T02:03:45.729 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.729 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.729 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:45.730 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:45.730 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:45.730 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:45.730 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 440 2024-04-19T02:03:45.730 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.730 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.730 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.730 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.730 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194253+0000", 2024-04-19T02:03:45.730 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.730 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.730 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.730 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.730 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194253+0000", 2024-04-19T02:03:45.731 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.731 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.731 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.731 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.731 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194254+0000", 2024-04-19T02:03:45.731 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8299999999999998e-07 2024-04-19T02:03:45.731 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.731 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.731 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.731 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194255+0000", 2024-04-19T02:03:45.731 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7120000000000001e-06 2024-04-19T02:03:45.731 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.731 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.731 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.732 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194259+0000", 2024-04-19T02:03:45.732 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.517e-06 2024-04-19T02:03:45.732 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.732 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.732 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.732 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194261+0000", 2024-04-19T02:03:45.732 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.689e-06 2024-04-19T02:03:45.732 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.732 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.732 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.732 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.205980+0000", 2024-04-19T02:03:45.732 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.011718561000000001 2024-04-19T02:03:45.732 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.733 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.733 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:45.733 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.206074+0000", 2024-04-19T02:03:45.733 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3556999999999999e-05 2024-04-19T02:03:45.733 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.733 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.733 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:45.733 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.206673+0000", 2024-04-19T02:03:45.733 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00059902100000000004 2024-04-19T02:03:45.733 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.733 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.733 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:45.733 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.208936+0000", 2024-04-19T02:03:45.734 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002262975 2024-04-19T02:03:45.734 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.734 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.734 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:45.734 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.236430+0000", 2024-04-19T02:03:45.734 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.027494338 2024-04-19T02:03:45.734 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.734 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.734 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.734 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.236487+0000", 2024-04-19T02:03:45.734 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7077000000000003e-05 2024-04-19T02:03:45.734 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.734 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.734 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.735 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.735 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.735 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:441 3.2s0 3:43ce42f3:::smithi05930959-27:head [append 0~0] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:45.735 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194270+0000", 2024-04-19T02:03:45.735 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 15.46536489, 2024-04-19T02:03:45.735 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.042541760999999997, 2024-04-19T02:03:45.735 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.735 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.735 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:45.735 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:45.735 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:45.735 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:45.735 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 441 2024-04-19T02:03:45.735 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.736 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.736 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.736 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.736 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194270+0000", 2024-04-19T02:03:45.736 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.736 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.736 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.736 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.736 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194270+0000", 2024-04-19T02:03:45.736 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.736 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.736 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.736 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.736 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194270+0000", 2024-04-19T02:03:45.736 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2500000000000001e-07 2024-04-19T02:03:45.737 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.737 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.737 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.737 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194271+0000", 2024-04-19T02:03:45.737 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3939999999999999e-06 2024-04-19T02:03:45.737 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.737 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.737 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.737 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194275+0000", 2024-04-19T02:03:45.737 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.827e-06 2024-04-19T02:03:45.737 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.737 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.737 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.737 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194278+0000", 2024-04-19T02:03:45.738 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7889999999999999e-06 2024-04-19T02:03:45.738 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.738 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.738 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.738 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.207160+0000", 2024-04-19T02:03:45.738 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012881867999999999 2024-04-19T02:03:45.738 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.738 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.738 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:45.738 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.207262+0000", 2024-04-19T02:03:45.738 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000102142 2024-04-19T02:03:45.738 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.738 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.738 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:45.739 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.207787+0000", 2024-04-19T02:03:45.739 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00052537099999999995 2024-04-19T02:03:45.739 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.739 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.739 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:45.739 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.208937+0000", 2024-04-19T02:03:45.739 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00114908 2024-04-19T02:03:45.739 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.739 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.739 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:45.739 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.236712+0000", 2024-04-19T02:03:45.739 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.027775428000000001 2024-04-19T02:03:45.739 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.740 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.740 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.740 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.236811+0000", 2024-04-19T02:03:45.740 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9537000000000006e-05 2024-04-19T02:03:45.740 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.740 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.740 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.740 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.740 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.740 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(osd.7.18:2 3.2s0 3:465821b6:::smithi05930959-32:head [assert-version v14,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:45.740 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.199189+0000", 2024-04-19T02:03:45.740 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 15.460445655000001, 2024-04-19T02:03:45.740 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.035632812, 2024-04-19T02:03:45.740 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.740 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.741 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:45.741 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.741 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.741 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.741 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.199189+0000", 2024-04-19T02:03:45.741 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.741 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.741 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.741 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.741 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.199189+0000", 2024-04-19T02:03:45.741 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.741 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.742 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.742 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.742 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.199190+0000", 2024-04-19T02:03:45.742 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5069999999999999e-06 2024-04-19T02:03:45.742 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.742 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.742 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.742 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.199198+0000", 2024-04-19T02:03:45.742 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.199e-06 2024-04-19T02:03:45.742 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.742 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.742 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.742 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.199211+0000", 2024-04-19T02:03:45.742 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3567000000000001e-05 2024-04-19T02:03:45.742 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.743 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.743 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.743 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.199220+0000", 2024-04-19T02:03:45.743 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2719999999999997e-06 2024-04-19T02:03:45.743 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.743 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.743 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.743 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.200485+0000", 2024-04-19T02:03:45.743 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001265195 2024-04-19T02:03:45.743 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.743 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.743 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:45.743 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.200596+0000", 2024-04-19T02:03:45.743 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00011098 2024-04-19T02:03:45.744 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.744 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.744 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.744 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.234822+0000", 2024-04-19T02:03:45.744 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.034226092 2024-04-19T02:03:45.744 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.744 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.744 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.744 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.744 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.744 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.2s0 46/26 ECSubReadReply(tid=166, attrs_read=0))", 2024-04-19T02:03:45.744 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.209587+0000", 2024-04-19T02:03:45.744 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 15.450047469999999, 2024-04-19T02:03:45.744 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.025249772, 2024-04-19T02:03:45.745 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.745 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.745 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:45.745 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.745 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.745 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.745 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.209587+0000", 2024-04-19T02:03:45.745 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.745 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.745 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.745 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.745 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.209587+0000", 2024-04-19T02:03:45.745 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.745 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.745 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.746 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.746 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.209588+0000", 2024-04-19T02:03:45.746 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.4099999999999999e-07 2024-04-19T02:03:45.746 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.746 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.746 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.746 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.209834+0000", 2024-04-19T02:03:45.746 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024650200000000001 2024-04-19T02:03:45.746 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.746 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.746 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.746 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.209854+0000", 2024-04-19T02:03:45.746 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9758999999999998e-05 2024-04-19T02:03:45.746 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.746 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.747 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.747 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.209867+0000", 2024-04-19T02:03:45.747 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2917e-05 2024-04-19T02:03:45.747 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.747 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.747 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.747 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.234535+0000", 2024-04-19T02:03:45.747 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.024668606999999999 2024-04-19T02:03:45.747 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.747 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.747 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.747 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.234837+0000", 2024-04-19T02:03:45.747 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00030144599999999997 2024-04-19T02:03:45.747 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.748 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.748 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.748 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.748 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.748 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 46/26 ECSubWriteReply(tid=163, last_complete=46'78, committed=1, applied=1))", 2024-04-19T02:03:45.748 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.210911+0000", 2024-04-19T02:03:45.748 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 15.448723478, 2024-04-19T02:03:45.748 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.024349902, 2024-04-19T02:03:45.748 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.748 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.748 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:45.748 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.748 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.748 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.748 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210911+0000", 2024-04-19T02:03:45.748 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.749 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.749 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.749 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.749 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210911+0000", 2024-04-19T02:03:45.749 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.749 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.749 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.749 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.749 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210912+0000", 2024-04-19T02:03:45.749 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8700000000000005e-07 2024-04-19T02:03:45.749 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.749 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.749 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.749 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210916+0000", 2024-04-19T02:03:45.749 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4340000000000002e-06 2024-04-19T02:03:45.749 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.750 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.750 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.750 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210933+0000", 2024-04-19T02:03:45.750 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7125999999999998e-05 2024-04-19T02:03:45.750 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.750 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.750 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.750 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210952+0000", 2024-04-19T02:03:45.750 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8320000000000001e-05 2024-04-19T02:03:45.750 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.750 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.750 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.750 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235246+0000", 2024-04-19T02:03:45.750 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.024294759999999999 2024-04-19T02:03:45.751 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.751 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.751 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.751 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235261+0000", 2024-04-19T02:03:45.751 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4575e-05 2024-04-19T02:03:45.751 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.751 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.751 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.751 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.751 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.751 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 46/26 ECSubWriteReply(tid=164, last_complete=46'79, committed=1, applied=1))", 2024-04-19T02:03:45.751 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.210983+0000", 2024-04-19T02:03:45.751 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 15.448651663, 2024-04-19T02:03:45.751 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.024646257000000001, 2024-04-19T02:03:45.751 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.752 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.752 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:45.752 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.752 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.752 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.752 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210983+0000", 2024-04-19T02:03:45.752 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.752 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.752 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.752 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.752 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210983+0000", 2024-04-19T02:03:45.752 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.752 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.752 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.752 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.753 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210983+0000", 2024-04-19T02:03:45.753 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2100000000000002e-07 2024-04-19T02:03:45.753 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.753 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.753 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.753 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210987+0000", 2024-04-19T02:03:45.753 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2789999999999999e-06 2024-04-19T02:03:45.753 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.753 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.753 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.753 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210995+0000", 2024-04-19T02:03:45.753 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3790000000000006e-06 2024-04-19T02:03:45.753 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.754 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.754 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.754 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.211007+0000", 2024-04-19T02:03:45.754 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2373e-05 2024-04-19T02:03:45.754 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.754 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.754 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.754 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235615+0000", 2024-04-19T02:03:45.754 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.024607688999999999 2024-04-19T02:03:45.754 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.754 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.754 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.754 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235629+0000", 2024-04-19T02:03:45.754 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4116e-05 2024-04-19T02:03:45.755 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.755 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.755 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.755 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.755 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.755 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 46/26 ECSubWriteReply(tid=167, last_complete=46'81, committed=1, applied=1))", 2024-04-19T02:03:45.755 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.211022+0000", 2024-04-19T02:03:45.755 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 15.448612777999999, 2024-04-19T02:03:45.755 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.024969574000000001, 2024-04-19T02:03:45.755 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:45.755 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:45.755 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:45.755 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:45.755 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.756 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:45.756 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.211022+0000", 2024-04-19T02:03:45.756 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.756 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.756 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.756 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:45.756 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.211022+0000", 2024-04-19T02:03:45.756 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:45.756 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.756 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.756 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:45.756 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.211022+0000", 2024-04-19T02:03:45.756 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2000000000000001e-07 2024-04-19T02:03:45.757 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.757 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.757 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:45.757 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.211025+0000", 2024-04-19T02:03:45.757 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.717e-06 2024-04-19T02:03:45.757 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.757 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.757 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:45.757 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.218661+0000", 2024-04-19T02:03:45.757 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0076366139999999999 2024-04-19T02:03:45.757 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.757 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.757 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:45.758 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.218669+0000", 2024-04-19T02:03:45.758 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.678e-06 2024-04-19T02:03:45.758 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.758 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.758 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:45.758 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235978+0000", 2024-04-19T02:03:45.758 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.017308430999999999 2024-04-19T02:03:45.758 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:45.758 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:45.758 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:45.758 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235991+0000", 2024-04-19T02:03:45.758 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3814e-05 2024-04-19T02:03:45.758 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.759 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.759 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.759 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:45.759 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:45.759 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:45.760 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-19T02:03:45.761 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:45.761+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:45.861 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:45.862+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:45.891 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:45.891 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:45.892 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 0 2024-04-19T02:03:45.892 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:45.905 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-19T02:03:45.962 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:45.961+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:46.028 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:46.028 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:46.028 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-04-19T02:03:46.028 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-04-19T02:03:46.028 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:46.043 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-19T02:03:46.063 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:46.061+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:46.164 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:46.165+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:46.182 INFO:tasks.rados.rados.0.smithi059.stdout:251: done (2 left) 2024-04-19T02:03:46.182 INFO:tasks.rados.rados.0.smithi059.stdout:252: done (1 left) 2024-04-19T02:03:46.182 INFO:tasks.rados.rados.0.smithi059.stdout:253: done (0 left) 2024-04-19T02:03:46.182 INFO:tasks.rados.rados.0.smithi059.stdout:254: read oid 28 snap -1 2024-04-19T02:03:46.182 INFO:tasks.rados.rados.0.smithi059.stdout:254: expect (ObjNum 27 snap 0 seq_num 27) 2024-04-19T02:03:46.182 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 43 current snap is 19 2024-04-19T02:03:46.182 INFO:tasks.rados.rados.0.smithi059.stdout:255: seq_num 120 ranges {1245184=475136,1720320=655360,2375680=622592,2998272=278528} 2024-04-19T02:03:46.203 INFO:tasks.rados.rados.0.smithi059.stdout:255: writing smithi05930959-43 from 1245184 to 1720320 tid 1 2024-04-19T02:03:46.213 INFO:tasks.rados.rados.0.smithi059.stdout:255: writing smithi05930959-43 from 1720320 to 2375680 tid 2 2024-04-19T02:03:46.214 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:46.214 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-04-19T02:03:46.214 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-04-19T02:03:46.214 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-04-19T02:03:46.214 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.215 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.es1 35/26 ECSubWrite(tid=23, reqid=osd.7.0:23, at_version=35'18, trim_to=0'0, roll_forward_to=34'16))", 2024-04-19T02:03:46.215 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:19.676794+0000", 2024-04-19T02:03:46.215 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 26.535912670999998, 2024-04-19T02:03:46.215 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012162914, 2024-04-19T02:03:46.215 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:46.215 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:46.215 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:46.215 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:46.215 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.215 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:46.215 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.676794+0000", 2024-04-19T02:03:46.215 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.215 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.215 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.216 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:46.216 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.676794+0000", 2024-04-19T02:03:46.216 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.216 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.216 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.216 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:46.216 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.676795+0000", 2024-04-19T02:03:46.216 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.04e-06 2024-04-19T02:03:46.216 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.216 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.216 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:46.217 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.677097+0000", 2024-04-19T02:03:46.217 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00030220099999999999 2024-04-19T02:03:46.217 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.217 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.217 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:46.217 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.677137+0000", 2024-04-19T02:03:46.217 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9756999999999999e-05 2024-04-19T02:03:46.217 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.217 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.217 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:46.217 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.677442+0000", 2024-04-19T02:03:46.217 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00030463799999999998 2024-04-19T02:03:46.217 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.217 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.218 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:46.218 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.677615+0000", 2024-04-19T02:03:46.218 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017371899999999999 2024-04-19T02:03:46.218 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.218 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.218 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:46.218 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.677641+0000", 2024-04-19T02:03:46.218 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5908e-05 2024-04-19T02:03:46.218 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.218 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.218 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:46.218 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.688872+0000", 2024-04-19T02:03:46.218 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011230957999999999 2024-04-19T02:03:46.218 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.218 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.219 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:46.219 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:19.688957+0000", 2024-04-19T02:03:46.219 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4692999999999997e-05 2024-04-19T02:03:46.219 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.219 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:46.219 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.219 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.219 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.219 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.cs4 46/26 ECSubWrite(tid=44, reqid=osd.9.0:44, at_version=46'26, trim_to=0'0, roll_forward_to=45'24))", 2024-04-19T02:03:46.219 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.065744+0000", 2024-04-19T02:03:46.219 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 16.146962653999999, 2024-04-19T02:03:46.219 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.018889659, 2024-04-19T02:03:46.219 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:46.219 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:46.219 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:46.220 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:46.220 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.220 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:46.220 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.065744+0000", 2024-04-19T02:03:46.220 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.220 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.220 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.220 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:46.220 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.065744+0000", 2024-04-19T02:03:46.220 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.221 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.221 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.221 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:46.221 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.065745+0000", 2024-04-19T02:03:46.221 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0999999999999998e-07 2024-04-19T02:03:46.221 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.221 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.221 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:46.221 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.065794+0000", 2024-04-19T02:03:46.221 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9177e-05 2024-04-19T02:03:46.221 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.222 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.222 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:46.222 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.065836+0000", 2024-04-19T02:03:46.222 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2245e-05 2024-04-19T02:03:46.222 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.222 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.222 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:46.222 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.065860+0000", 2024-04-19T02:03:46.222 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3654000000000001e-05 2024-04-19T02:03:46.222 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.222 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.222 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:46.222 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066138+0000", 2024-04-19T02:03:46.222 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00027793700000000002 2024-04-19T02:03:46.223 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.223 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.223 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:46.223 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066164+0000", 2024-04-19T02:03:46.223 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5862000000000001e-05 2024-04-19T02:03:46.223 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.223 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.223 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:46.223 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.084533+0000", 2024-04-19T02:03:46.223 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.018369534 2024-04-19T02:03:46.223 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.223 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.223 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:46.223 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.084634+0000", 2024-04-19T02:03:46.223 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010054 2024-04-19T02:03:46.224 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.224 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:46.224 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.224 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.224 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.224 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:431 3.8s0 3:1984c4bc:::smithi05930959-40:head [read 0~3162112,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:46.224 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.193642+0000", 2024-04-19T02:03:46.224 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 16.019064301, 2024-04-19T02:03:46.224 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.015491476000000001, 2024-04-19T02:03:46.224 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:46.224 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:46.224 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:03:46.224 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:46.224 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:46.224 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:46.224 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 431 2024-04-19T02:03:46.225 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.225 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:46.225 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.225 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:46.225 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193642+0000", 2024-04-19T02:03:46.225 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.225 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.225 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.225 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:46.225 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193642+0000", 2024-04-19T02:03:46.225 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.225 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.225 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.225 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:46.225 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193645+0000", 2024-04-19T02:03:46.226 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0369999999999999e-06 2024-04-19T02:03:46.226 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.226 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.226 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:46.226 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193659+0000", 2024-04-19T02:03:46.226 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3424999999999999e-05 2024-04-19T02:03:46.226 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.226 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.226 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:46.226 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193692+0000", 2024-04-19T02:03:46.226 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3382e-05 2024-04-19T02:03:46.226 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.226 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.226 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:46.227 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193714+0000", 2024-04-19T02:03:46.227 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1522000000000002e-05 2024-04-19T02:03:46.227 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.227 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.227 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:46.227 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193838+0000", 2024-04-19T02:03:46.227 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012381499999999999 2024-04-19T02:03:46.227 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.227 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.227 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:46.227 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.194001+0000", 2024-04-19T02:03:46.227 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016321 2024-04-19T02:03:46.227 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.227 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.227 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:46.227 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.209134+0000", 2024-04-19T02:03:46.228 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.015133084999999999 2024-04-19T02:03:46.228 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.228 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:46.228 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.228 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.228 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.228 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:432 3.8s0 3:1984c4bc:::smithi05930959-40:head [read 0~3162112,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:46.228 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.193759+0000", 2024-04-19T02:03:46.228 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 16.018947471000001, 2024-04-19T02:03:46.228 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.020892661, 2024-04-19T02:03:46.228 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:46.228 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:46.228 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:03:46.228 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:46.228 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:46.229 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:46.229 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 432 2024-04-19T02:03:46.229 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.229 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:46.229 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.229 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:46.229 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193759+0000", 2024-04-19T02:03:46.229 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.229 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.229 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.229 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:46.229 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193759+0000", 2024-04-19T02:03:46.229 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.229 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.229 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.230 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:46.230 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193761+0000", 2024-04-19T02:03:46.230 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3430000000000001e-06 2024-04-19T02:03:46.230 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.230 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.230 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:46.230 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193770+0000", 2024-04-19T02:03:46.230 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5359999999999998e-06 2024-04-19T02:03:46.230 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.230 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.230 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:46.230 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193794+0000", 2024-04-19T02:03:46.230 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3589000000000001e-05 2024-04-19T02:03:46.230 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.230 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.231 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:46.231 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193811+0000", 2024-04-19T02:03:46.231 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6838999999999999e-05 2024-04-19T02:03:46.231 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.231 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.231 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:46.231 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.194657+0000", 2024-04-19T02:03:46.231 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00084614800000000002 2024-04-19T02:03:46.231 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.231 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.231 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:46.231 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.194925+0000", 2024-04-19T02:03:46.231 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000267927 2024-04-19T02:03:46.231 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.232 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.232 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:46.232 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.214652+0000", 2024-04-19T02:03:46.232 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.019727279 2024-04-19T02:03:46.232 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.232 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:46.232 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.232 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.232 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.232 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:433 3.8s0 3:1984c4bc:::smithi05930959-40:head [read 0~3162112,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:46.232 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.193839+0000", 2024-04-19T02:03:46.232 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 16.018868052999998, 2024-04-19T02:03:46.232 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.025594295, 2024-04-19T02:03:46.232 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:46.232 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:46.232 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:03:46.233 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:46.233 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:46.233 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:46.233 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 433 2024-04-19T02:03:46.233 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.233 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:46.233 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.234 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:46.234 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193839+0000", 2024-04-19T02:03:46.234 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.234 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.234 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.234 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:46.234 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193839+0000", 2024-04-19T02:03:46.234 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.234 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.234 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.235 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:46.235 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193840+0000", 2024-04-19T02:03:46.235 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.277e-06 2024-04-19T02:03:46.235 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.235 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.235 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:46.235 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193848+0000", 2024-04-19T02:03:46.235 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0790000000000001e-06 2024-04-19T02:03:46.235 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.235 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.235 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:46.235 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193866+0000", 2024-04-19T02:03:46.235 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8411999999999999e-05 2024-04-19T02:03:46.235 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.236 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.236 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:46.236 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193881+0000", 2024-04-19T02:03:46.236 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4419e-05 2024-04-19T02:03:46.236 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.236 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.236 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:46.236 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.202091+0000", 2024-04-19T02:03:46.236 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0082103079999999995 2024-04-19T02:03:46.236 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.236 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.236 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:46.236 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.202540+0000", 2024-04-19T02:03:46.236 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00044865600000000002 2024-04-19T02:03:46.236 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.237 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.237 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:46.237 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.219433+0000", 2024-04-19T02:03:46.237 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.016893143999999999 2024-04-19T02:03:46.237 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.237 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:46.237 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.237 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.237 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.237 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.bs4 47/26 ECSubWrite(tid=180, reqid=osd.3.0:180, at_version=47'36, trim_to=0'0, roll_forward_to=46'34))", 2024-04-19T02:03:46.237 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.080143+0000", 2024-04-19T02:03:46.237 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 15.132563908, 2024-04-19T02:03:46.237 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012272063, 2024-04-19T02:03:46.237 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:46.237 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:46.237 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:46.238 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:46.238 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.238 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:46.238 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.080143+0000", 2024-04-19T02:03:46.238 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.238 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.238 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.238 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:46.238 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.080143+0000", 2024-04-19T02:03:46.238 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.239 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.239 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.239 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:46.239 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.080144+0000", 2024-04-19T02:03:46.240 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.266e-06 2024-04-19T02:03:46.240 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.240 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.240 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:46.240 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.080191+0000", 2024-04-19T02:03:46.240 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6693999999999997e-05 2024-04-19T02:03:46.241 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.241 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.241 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:46.241 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081012+0000", 2024-04-19T02:03:46.241 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00082073400000000005 2024-04-19T02:03:46.241 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.241 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.242 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:46.242 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081033+0000", 2024-04-19T02:03:46.242 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1648999999999999e-05 2024-04-19T02:03:46.242 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.242 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.242 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:46.242 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081209+0000", 2024-04-19T02:03:46.242 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017594 2024-04-19T02:03:46.242 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.242 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.242 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:46.243 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081243+0000", 2024-04-19T02:03:46.243 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3976999999999997e-05 2024-04-19T02:03:46.243 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.243 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.243 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:46.243 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.092330+0000", 2024-04-19T02:03:46.243 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011086972 2024-04-19T02:03:46.244 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.244 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.244 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:46.244 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.092415+0000", 2024-04-19T02:03:46.244 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4831000000000001e-05 2024-04-19T02:03:46.244 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.244 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:46.244 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.244 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.244 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.244 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.bs4 47/26 ECSubWrite(tid=183, reqid=osd.3.0:183, at_version=47'38, trim_to=0'0, roll_forward_to=46'34))", 2024-04-19T02:03:46.244 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.081558+0000", 2024-04-19T02:03:46.244 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 15.131148386, 2024-04-19T02:03:46.244 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014283580000000001, 2024-04-19T02:03:46.244 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:46.244 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:46.245 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:46.245 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:46.245 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.245 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:46.245 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081558+0000", 2024-04-19T02:03:46.245 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.245 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.245 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.245 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:46.245 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081558+0000", 2024-04-19T02:03:46.246 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.246 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.246 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.246 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:46.246 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081559+0000", 2024-04-19T02:03:46.246 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4300000000000001e-07 2024-04-19T02:03:46.246 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.246 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.246 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:46.246 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081613+0000", 2024-04-19T02:03:46.246 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3560000000000002e-05 2024-04-19T02:03:46.246 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.247 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.247 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:46.247 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081636+0000", 2024-04-19T02:03:46.247 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3266000000000001e-05 2024-04-19T02:03:46.247 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.247 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.247 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:46.247 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081653+0000", 2024-04-19T02:03:46.247 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6821999999999999e-05 2024-04-19T02:03:46.247 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.247 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.247 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:46.247 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.082251+0000", 2024-04-19T02:03:46.247 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00059773699999999999 2024-04-19T02:03:46.248 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.248 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.248 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:46.248 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.082276+0000", 2024-04-19T02:03:46.248 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5066e-05 2024-04-19T02:03:46.248 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.248 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.248 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:46.248 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.095735+0000", 2024-04-19T02:03:46.248 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013459575999999999 2024-04-19T02:03:46.248 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.248 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.248 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:46.248 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.095842+0000", 2024-04-19T02:03:46.249 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010661 2024-04-19T02:03:46.249 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.249 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:46.249 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.249 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.249 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.249 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.ds3 47/26 ECSubWrite(tid=193, reqid=osd.8.0:193, at_version=47'40, trim_to=0'0, roll_forward_to=46'38))", 2024-04-19T02:03:46.249 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.083005+0000", 2024-04-19T02:03:46.249 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 15.129702039, 2024-04-19T02:03:46.249 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013056883, 2024-04-19T02:03:46.249 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:46.249 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:46.249 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:46.249 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:46.249 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.249 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:46.250 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.083005+0000", 2024-04-19T02:03:46.250 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.250 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.250 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.250 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:46.250 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.083005+0000", 2024-04-19T02:03:46.250 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.250 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.250 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.250 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:46.250 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.083005+0000", 2024-04-19T02:03:46.250 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9299999999999998e-07 2024-04-19T02:03:46.250 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.250 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.251 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:46.251 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.083052+0000", 2024-04-19T02:03:46.251 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7036999999999997e-05 2024-04-19T02:03:46.251 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.251 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.251 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:46.251 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.083082+0000", 2024-04-19T02:03:46.251 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9451999999999999e-05 2024-04-19T02:03:46.251 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.251 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.251 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:46.251 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.083110+0000", 2024-04-19T02:03:46.251 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8317999999999999e-05 2024-04-19T02:03:46.251 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.251 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.252 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:46.252 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.083268+0000", 2024-04-19T02:03:46.252 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015767099999999999 2024-04-19T02:03:46.252 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.252 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.252 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:46.252 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.083293+0000", 2024-04-19T02:03:46.252 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5408000000000001e-05 2024-04-19T02:03:46.252 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.252 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.252 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:46.252 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.095974+0000", 2024-04-19T02:03:46.252 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012681289 2024-04-19T02:03:46.252 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.252 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.253 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:46.253 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.096062+0000", 2024-04-19T02:03:46.253 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7114999999999995e-05 2024-04-19T02:03:46.253 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.253 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:46.253 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.253 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.253 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.253 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.1s2 47/26 ECSubWrite(tid=30, reqid=osd.0.0:30, at_version=47'17, trim_to=0'0, roll_forward_to=46'13))", 2024-04-19T02:03:46.253 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.086105+0000", 2024-04-19T02:03:46.253 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 15.126601578000001, 2024-04-19T02:03:46.253 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013306735, 2024-04-19T02:03:46.253 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:46.253 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:46.254 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:46.254 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:46.254 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.254 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:46.254 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.086105+0000", 2024-04-19T02:03:46.254 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.254 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.254 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.254 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:46.254 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.086105+0000", 2024-04-19T02:03:46.254 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.254 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.254 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.254 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:46.255 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.086106+0000", 2024-04-19T02:03:46.255 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.2699999999999998e-07 2024-04-19T02:03:46.255 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.255 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.255 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:46.255 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.086151+0000", 2024-04-19T02:03:46.255 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5256999999999997e-05 2024-04-19T02:03:46.255 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.255 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.255 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:46.255 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.086175+0000", 2024-04-19T02:03:46.255 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3805999999999999e-05 2024-04-19T02:03:46.255 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.255 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.255 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:46.256 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.086192+0000", 2024-04-19T02:03:46.256 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7175000000000001e-05 2024-04-19T02:03:46.256 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.256 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.256 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:46.256 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.089760+0000", 2024-04-19T02:03:46.256 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0035675569999999998 2024-04-19T02:03:46.256 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.256 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.256 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:46.256 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.089783+0000", 2024-04-19T02:03:46.256 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3575e-05 2024-04-19T02:03:46.256 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.256 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.256 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:46.257 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.099331+0000", 2024-04-19T02:03:46.257 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0095480470000000005 2024-04-19T02:03:46.257 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.257 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.257 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:46.257 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.099412+0000", 2024-04-19T02:03:46.257 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0390999999999997e-05 2024-04-19T02:03:46.257 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.257 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:46.257 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.257 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.257 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.258 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:470 3.9s0 3:97177082:::smithi05930959-45:head [append 0~802816] snapc a=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:03:46.258 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.167632+0000", 2024-04-19T02:03:46.258 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 15.04507456, 2024-04-19T02:03:46.258 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011924591999999999, 2024-04-19T02:03:46.258 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:46.258 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:46.258 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:46.258 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:46.258 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:46.258 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:46.258 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 470 2024-04-19T02:03:46.258 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.259 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:46.259 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.259 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:46.259 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.167632+0000", 2024-04-19T02:03:46.259 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.259 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.259 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.259 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:46.259 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.167632+0000", 2024-04-19T02:03:46.259 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.259 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.259 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.259 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:46.259 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.167636+0000", 2024-04-19T02:03:46.260 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5200000000000002e-06 2024-04-19T02:03:46.260 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.260 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.260 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:46.260 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.169434+0000", 2024-04-19T02:03:46.260 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0017987890000000001 2024-04-19T02:03:46.260 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.260 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.260 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:46.260 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.169461+0000", 2024-04-19T02:03:46.260 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6035999999999999e-05 2024-04-19T02:03:46.260 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.260 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.261 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:46.261 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.169479+0000", 2024-04-19T02:03:46.261 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8241999999999999e-05 2024-04-19T02:03:46.261 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.261 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.261 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:46.261 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.169620+0000", 2024-04-19T02:03:46.261 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014108100000000001 2024-04-19T02:03:46.261 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.261 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.261 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:46.261 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.169850+0000", 2024-04-19T02:03:46.261 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000229702 2024-04-19T02:03:46.261 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.262 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.262 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:46.262 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.173203+0000", 2024-04-19T02:03:46.262 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0033530970000000002 2024-04-19T02:03:46.262 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.262 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.262 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:46.262 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.178706+0000", 2024-04-19T02:03:46.262 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0055031979999999999 2024-04-19T02:03:46.262 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.262 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.262 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:46.262 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.179377+0000", 2024-04-19T02:03:46.262 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00067095999999999996 2024-04-19T02:03:46.262 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.263 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.263 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:46.263 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.179557+0000", 2024-04-19T02:03:46.263 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017996700000000001 2024-04-19T02:03:46.263 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.263 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:46.263 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.263 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.263 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.263 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:473 3.9s0 3:97177082:::smithi05930959-45:head [setxattr _header (56) in=7b] snapc a=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:03:46.263 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.174085+0000", 2024-04-19T02:03:46.263 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 15.038621413, 2024-04-19T02:03:46.263 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012468714000000001, 2024-04-19T02:03:46.263 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:46.263 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:46.264 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:46.264 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:46.264 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:46.264 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:46.264 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 473 2024-04-19T02:03:46.264 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.264 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:46.264 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.264 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:46.264 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.174085+0000", 2024-04-19T02:03:46.264 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.264 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.264 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.264 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:46.264 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.174085+0000", 2024-04-19T02:03:46.265 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.265 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.265 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.265 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:46.265 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.174087+0000", 2024-04-19T02:03:46.265 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.567e-06 2024-04-19T02:03:46.265 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.265 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.265 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:46.265 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.174096+0000", 2024-04-19T02:03:46.265 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8929999999999994e-06 2024-04-19T02:03:46.265 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.265 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.265 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:46.266 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.174115+0000", 2024-04-19T02:03:46.266 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9445000000000001e-05 2024-04-19T02:03:46.266 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.266 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.266 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:46.266 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.174132+0000", 2024-04-19T02:03:46.266 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6651999999999999e-05 2024-04-19T02:03:46.266 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.266 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.266 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:46.266 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.182528+0000", 2024-04-19T02:03:46.266 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0083963030000000008 2024-04-19T02:03:46.266 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.267 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.267 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:46.267 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.182781+0000", 2024-04-19T02:03:46.267 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00025269700000000002 2024-04-19T02:03:46.267 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.267 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.267 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:46.267 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.183985+0000", 2024-04-19T02:03:46.267 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0012036530000000001 2024-04-19T02:03:46.267 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.267 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.267 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:46.267 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.185647+0000", 2024-04-19T02:03:46.267 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001662941 2024-04-19T02:03:46.268 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.268 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.268 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:46.268 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.186412+0000", 2024-04-19T02:03:46.268 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00076417200000000003 2024-04-19T02:03:46.268 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.268 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.268 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:46.268 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.186554+0000", 2024-04-19T02:03:46.268 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014239100000000001 2024-04-19T02:03:46.268 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.268 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:46.268 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.269 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.269 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.269 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:474 3.9s0 3:97177082:::smithi05930959-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:03:46.269 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.174152+0000", 2024-04-19T02:03:46.269 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 15.038554962999999, 2024-04-19T02:03:46.269 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.016567683999999999, 2024-04-19T02:03:46.269 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:46.269 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:46.269 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:03:46.270 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:46.270 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:46.270 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:46.270 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 474 2024-04-19T02:03:46.270 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.271 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:46.271 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.271 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:46.271 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.174152+0000", 2024-04-19T02:03:46.271 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.271 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.271 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.271 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:46.271 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.174152+0000", 2024-04-19T02:03:46.271 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.271 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.271 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.271 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:46.271 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.174153+0000", 2024-04-19T02:03:46.271 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.39e-06 2024-04-19T02:03:46.272 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.272 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.272 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:46.272 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.174159+0000", 2024-04-19T02:03:46.272 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4759999999999997e-06 2024-04-19T02:03:46.272 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.272 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.272 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:46.272 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.174179+0000", 2024-04-19T02:03:46.272 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9966000000000002e-05 2024-04-19T02:03:46.272 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.273 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.273 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:46.273 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.174194+0000", 2024-04-19T02:03:46.273 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5332000000000002e-05 2024-04-19T02:03:46.273 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.273 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.273 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:46.273 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.187276+0000", 2024-04-19T02:03:46.273 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013081603000000001 2024-04-19T02:03:46.273 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.273 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.273 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:46.273 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.187522+0000", 2024-04-19T02:03:46.273 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00024612499999999998 2024-04-19T02:03:46.273 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.274 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.274 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:46.274 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.190719+0000", 2024-04-19T02:03:46.274 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0031977920000000001 2024-04-19T02:03:46.274 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.274 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:46.274 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.274 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.274 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.274 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:576 3.8s0 3:16258ed3:::smithi05930959-3:head [append 0~655360] snapc 10=[10,f,c] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-19T02:03:46.274 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:40.161424+0000", 2024-04-19T02:03:46.274 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 6.0512824549999999, 2024-04-19T02:03:46.274 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.018735465, 2024-04-19T02:03:46.275 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:46.275 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:46.275 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:46.275 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:46.275 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:46.275 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:46.275 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 576 2024-04-19T02:03:46.275 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.275 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:46.275 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.275 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:46.275 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.161424+0000", 2024-04-19T02:03:46.275 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.275 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.275 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.276 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:46.276 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.161424+0000", 2024-04-19T02:03:46.276 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.276 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.276 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.276 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:46.276 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.161428+0000", 2024-04-19T02:03:46.276 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8670000000000004e-06 2024-04-19T02:03:46.276 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.276 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.276 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:46.276 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.162988+0000", 2024-04-19T02:03:46.276 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001559476 2024-04-19T02:03:46.276 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.276 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.276 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:46.277 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.163028+0000", 2024-04-19T02:03:46.277 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9950999999999997e-05 2024-04-19T02:03:46.277 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.277 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.277 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:46.277 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.163047+0000", 2024-04-19T02:03:46.277 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9162000000000001e-05 2024-04-19T02:03:46.277 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.277 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.277 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:46.277 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.163203+0000", 2024-04-19T02:03:46.277 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015594 2024-04-19T02:03:46.277 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.277 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.277 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op must wait for map", 2024-04-19T02:03:46.278 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.163215+0000", 2024-04-19T02:03:46.278 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2748999999999999e-05 2024-04-19T02:03:46.278 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.278 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.278 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:46.278 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.166369+0000", 2024-04-19T02:03:46.278 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0031538080000000001 2024-04-19T02:03:46.278 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.278 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.278 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:46.278 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.166492+0000", 2024-04-19T02:03:46.278 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012248900000000001 2024-04-19T02:03:46.278 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.278 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.278 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:46.279 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.168148+0000", 2024-04-19T02:03:46.279 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0016566809999999999 2024-04-19T02:03:46.279 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.279 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.279 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:46.279 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.170497+0000", 2024-04-19T02:03:46.279 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0023481830000000002 2024-04-19T02:03:46.279 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.279 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.279 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:46.279 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.180068+0000", 2024-04-19T02:03:46.279 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0095713729999999993 2024-04-19T02:03:46.279 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.279 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.279 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:46.280 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.180160+0000", 2024-04-19T02:03:46.280 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1786000000000005e-05 2024-04-19T02:03:46.280 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.280 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:46.280 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.280 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.280 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.280 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:577 3.8s0 3:16258ed3:::smithi05930959-3:head [append 0~606208] snapc 10=[10,f,c] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-19T02:03:46.280 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:40.168979+0000", 2024-04-19T02:03:46.280 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 6.0437279500000001, 2024-04-19T02:03:46.280 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012599301, 2024-04-19T02:03:46.280 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:46.280 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:46.280 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:46.280 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:46.281 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:46.281 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:46.281 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 577 2024-04-19T02:03:46.281 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.281 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:46.281 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.281 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:46.281 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.168979+0000", 2024-04-19T02:03:46.281 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.281 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.281 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.281 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:46.281 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.168979+0000", 2024-04-19T02:03:46.281 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.282 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.282 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.282 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:46.282 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.168988+0000", 2024-04-19T02:03:46.282 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.9199999999999993e-06 2024-04-19T02:03:46.282 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.282 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.282 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:46.282 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.170972+0000", 2024-04-19T02:03:46.282 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0019847630000000001 2024-04-19T02:03:46.282 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.282 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.282 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:46.282 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.171009+0000", 2024-04-19T02:03:46.282 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6838000000000002e-05 2024-04-19T02:03:46.283 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.283 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.283 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:46.283 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.171034+0000", 2024-04-19T02:03:46.283 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4739000000000001e-05 2024-04-19T02:03:46.283 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.283 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.283 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:46.283 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.171130+0000", 2024-04-19T02:03:46.283 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5939000000000004e-05 2024-04-19T02:03:46.283 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.283 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.283 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:46.283 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.171245+0000", 2024-04-19T02:03:46.283 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000115349 2024-04-19T02:03:46.284 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.284 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.284 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:46.284 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.172440+0000", 2024-04-19T02:03:46.284 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001194943 2024-04-19T02:03:46.284 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.284 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.284 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:46.284 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.174609+0000", 2024-04-19T02:03:46.284 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0021686100000000001 2024-04-19T02:03:46.284 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.284 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.284 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:46.284 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.181443+0000", 2024-04-19T02:03:46.284 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0068343579999999996 2024-04-19T02:03:46.285 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.285 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.285 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:46.285 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.181578+0000", 2024-04-19T02:03:46.285 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013484200000000001 2024-04-19T02:03:46.285 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.285 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:46.285 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.285 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.285 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.285 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:581 3.8s0 3:16258ed3:::smithi05930959-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56)", 2024-04-19T02:03:46.285 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:40.176936+0000", 2024-04-19T02:03:46.285 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 6.0357703770000004, 2024-04-19T02:03:46.285 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012536441000000001, 2024-04-19T02:03:46.285 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:46.286 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:46.286 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:03:46.286 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:46.286 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:46.286 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:46.286 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 581 2024-04-19T02:03:46.286 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.286 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:46.286 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.286 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:46.286 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.176936+0000", 2024-04-19T02:03:46.286 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.286 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.286 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.286 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:46.287 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.176936+0000", 2024-04-19T02:03:46.287 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.287 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.287 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.287 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:46.287 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.176937+0000", 2024-04-19T02:03:46.287 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5000000000000001e-07 2024-04-19T02:03:46.287 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.287 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.287 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:46.287 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.176940+0000", 2024-04-19T02:03:46.287 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1130000000000001e-06 2024-04-19T02:03:46.287 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.287 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.287 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:46.288 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.176951+0000", 2024-04-19T02:03:46.288 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0723000000000001e-05 2024-04-19T02:03:46.288 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.288 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.288 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:46.288 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.176959+0000", 2024-04-19T02:03:46.288 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7800000000000001e-06 2024-04-19T02:03:46.288 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.288 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.288 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:46.288 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.181244+0000", 2024-04-19T02:03:46.288 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0042850010000000001 2024-04-19T02:03:46.288 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.288 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.288 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:46.288 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.181325+0000", 2024-04-19T02:03:46.289 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0927999999999995e-05 2024-04-19T02:03:46.289 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.289 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.289 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:46.289 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.186287+0000", 2024-04-19T02:03:46.289 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0049624980000000001 2024-04-19T02:03:46.289 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.289 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.289 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:46.289 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.187163+0000", 2024-04-19T02:03:46.289 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00087595399999999997 2024-04-19T02:03:46.289 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.289 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.289 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:46.289 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.189473+0000", 2024-04-19T02:03:46.290 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0023095939999999999 2024-04-19T02:03:46.290 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.290 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:46.290 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.290 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.290 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.290 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:635 3.8s0 3:1984c4bc:::smithi05930959-40:head [read 0~1490944,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-19T02:03:46.290 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:45.184636+0000", 2024-04-19T02:03:46.290 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.028070832, 2024-04-19T02:03:46.290 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012307169, 2024-04-19T02:03:46.290 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:46.290 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:46.290 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:03:46.290 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:46.290 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:46.291 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:46.291 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 635 2024-04-19T02:03:46.291 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.291 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:46.291 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.291 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:46.291 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.184636+0000", 2024-04-19T02:03:46.291 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.291 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.291 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.291 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:46.291 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.184636+0000", 2024-04-19T02:03:46.291 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.291 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.291 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.292 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:46.292 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.184639+0000", 2024-04-19T02:03:46.292 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.207e-06 2024-04-19T02:03:46.292 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.292 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.292 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:46.292 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.184655+0000", 2024-04-19T02:03:46.292 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5956000000000001e-05 2024-04-19T02:03:46.292 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.292 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.292 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:46.292 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.184694+0000", 2024-04-19T02:03:46.292 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8818e-05 2024-04-19T02:03:46.292 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.292 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.292 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:46.293 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.184721+0000", 2024-04-19T02:03:46.293 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7256000000000001e-05 2024-04-19T02:03:46.293 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.293 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.293 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:46.293 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.184875+0000", 2024-04-19T02:03:46.293 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015407100000000001 2024-04-19T02:03:46.293 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.293 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.293 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op must wait for map", 2024-04-19T02:03:46.293 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.184888+0000", 2024-04-19T02:03:46.293 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3275000000000001e-05 2024-04-19T02:03:46.293 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.293 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.293 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:46.294 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.188824+0000", 2024-04-19T02:03:46.294 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003935717 2024-04-19T02:03:46.294 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.294 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.294 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:46.294 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.188979+0000", 2024-04-19T02:03:46.294 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015463 2024-04-19T02:03:46.294 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.294 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.294 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:46.294 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.196943+0000", 2024-04-19T02:03:46.294 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0079642389999999997 2024-04-19T02:03:46.294 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.294 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:46.294 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.295 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.295 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.295 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:636 3.8s0 3:1984c4bc:::smithi05930959-40:head [read 0~1490944,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-19T02:03:46.295 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:45.184814+0000", 2024-04-19T02:03:46.295 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.0278922589999999, 2024-04-19T02:03:46.295 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013906419999999999, 2024-04-19T02:03:46.295 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:46.295 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:46.295 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:03:46.295 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:46.295 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:46.295 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:46.295 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 636 2024-04-19T02:03:46.295 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.296 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:46.296 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.296 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:46.296 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.184814+0000", 2024-04-19T02:03:46.296 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.296 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.296 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.296 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:46.296 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.184814+0000", 2024-04-19T02:03:46.296 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.296 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.296 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.296 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:46.296 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.184816+0000", 2024-04-19T02:03:46.296 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.007e-06 2024-04-19T02:03:46.296 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.297 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.297 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:46.297 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.184830+0000", 2024-04-19T02:03:46.297 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3123e-05 2024-04-19T02:03:46.297 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.297 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.297 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:46.297 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.184858+0000", 2024-04-19T02:03:46.297 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7965000000000001e-05 2024-04-19T02:03:46.297 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.297 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.297 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:46.297 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.184878+0000", 2024-04-19T02:03:46.297 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0939000000000001e-05 2024-04-19T02:03:46.297 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.298 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.298 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:46.298 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.185023+0000", 2024-04-19T02:03:46.298 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000144838 2024-04-19T02:03:46.298 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.298 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.298 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:03:46.298 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.185051+0000", 2024-04-19T02:03:46.298 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7915e-05 2024-04-19T02:03:46.298 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.298 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.298 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:46.298 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.189703+0000", 2024-04-19T02:03:46.298 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0046516309999999998 2024-04-19T02:03:46.298 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.299 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.299 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:46.299 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.189880+0000", 2024-04-19T02:03:46.299 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017668600000000001 2024-04-19T02:03:46.299 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.299 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.299 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:46.299 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.198721+0000", 2024-04-19T02:03:46.299 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0088413160000000001 2024-04-19T02:03:46.299 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.299 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:46.299 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.299 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.299 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.299 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:637 3.8s0 3:1984c4bc:::smithi05930959-40:head [read 0~1490944,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-19T02:03:46.300 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:45.184908+0000", 2024-04-19T02:03:46.300 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 1.0277991989999999, 2024-04-19T02:03:46.300 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.017005276999999999, 2024-04-19T02:03:46.300 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:46.300 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:46.300 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:03:46.300 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:46.300 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:46.300 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:46.300 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 637 2024-04-19T02:03:46.300 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.300 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:46.300 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.300 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:46.300 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.184908+0000", 2024-04-19T02:03:46.301 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.301 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.301 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.301 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:46.301 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.184908+0000", 2024-04-19T02:03:46.301 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.301 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.301 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.301 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:46.301 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.184909+0000", 2024-04-19T02:03:46.301 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5629999999999999e-06 2024-04-19T02:03:46.301 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.301 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.301 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:46.301 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.184918+0000", 2024-04-19T02:03:46.302 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8259999999999999e-06 2024-04-19T02:03:46.302 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.302 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.302 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:46.302 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.184938+0000", 2024-04-19T02:03:46.302 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0463e-05 2024-04-19T02:03:46.302 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.302 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.302 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:46.302 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.184954+0000", 2024-04-19T02:03:46.302 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5353e-05 2024-04-19T02:03:46.302 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.302 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.302 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:46.302 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.185127+0000", 2024-04-19T02:03:46.303 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017363900000000001 2024-04-19T02:03:46.303 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.303 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.303 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:03:46.303 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.185145+0000", 2024-04-19T02:03:46.303 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8116000000000001e-05 2024-04-19T02:03:46.303 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.303 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.303 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:46.303 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.193765+0000", 2024-04-19T02:03:46.303 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0086199459999999999 2024-04-19T02:03:46.303 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.303 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.303 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:46.304 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.194071+0000", 2024-04-19T02:03:46.304 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00030512399999999999 2024-04-19T02:03:46.304 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.304 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.304 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:46.304 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.201913+0000", 2024-04-19T02:03:46.304 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0078422470000000001 2024-04-19T02:03:46.304 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.304 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:46.304 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.304 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.304 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.304 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:640 3.9s0 3:90642b28:::smithi05930959-8:head [append 0~507904] snapc 13=[13,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-19T02:03:46.304 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:45.215757+0000", 2024-04-19T02:03:46.304 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.99694987599999996, 2024-04-19T02:03:46.304 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.01278, 2024-04-19T02:03:46.305 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:46.305 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:46.305 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:46.305 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:46.305 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:46.305 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:46.305 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 640 2024-04-19T02:03:46.305 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.305 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:46.305 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.305 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:46.305 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.215757+0000", 2024-04-19T02:03:46.305 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.305 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.305 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.306 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:46.306 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.215757+0000", 2024-04-19T02:03:46.306 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.306 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.306 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.306 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:46.306 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.215758+0000", 2024-04-19T02:03:46.306 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.604e-06 2024-04-19T02:03:46.306 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.306 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.306 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:46.306 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.216444+0000", 2024-04-19T02:03:46.306 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00068511799999999997 2024-04-19T02:03:46.306 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.306 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.307 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:46.307 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.216474+0000", 2024-04-19T02:03:46.307 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9963000000000001e-05 2024-04-19T02:03:46.307 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.307 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.307 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:46.307 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.216495+0000", 2024-04-19T02:03:46.307 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1104999999999999e-05 2024-04-19T02:03:46.307 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.307 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.307 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:46.307 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.216857+0000", 2024-04-19T02:03:46.307 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00036236200000000002 2024-04-19T02:03:46.307 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.307 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.308 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:46.308 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.217013+0000", 2024-04-19T02:03:46.308 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015581699999999999 2024-04-19T02:03:46.308 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.308 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.308 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:46.308 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.219833+0000", 2024-04-19T02:03:46.308 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0028203149999999999 2024-04-19T02:03:46.308 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.308 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.308 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:46.308 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.222887+0000", 2024-04-19T02:03:46.308 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0030534759999999998 2024-04-19T02:03:46.308 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.308 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.308 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:03:46.309 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.228413+0000", 2024-04-19T02:03:46.309 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0055262920000000004 2024-04-19T02:03:46.309 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.309 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.309 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:46.309 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.228537+0000", 2024-04-19T02:03:46.309 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000123948 2024-04-19T02:03:46.309 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.309 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:46.309 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.309 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.309 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.309 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:644 3.9s0 3:90642b28:::smithi05930959-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e61)", 2024-04-19T02:03:46.309 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:45.222718+0000", 2024-04-19T02:03:46.310 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.98998837500000003, 2024-04-19T02:03:46.310 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014793270000000001, 2024-04-19T02:03:46.310 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:46.310 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:46.310 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:03:46.310 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:03:46.310 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:03:46.310 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:46.310 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 644 2024-04-19T02:03:46.310 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.310 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:46.310 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.310 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:46.310 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.222718+0000", 2024-04-19T02:03:46.310 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.311 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.311 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.311 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:46.311 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.222718+0000", 2024-04-19T02:03:46.311 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:46.311 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.311 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.311 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:46.311 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.222719+0000", 2024-04-19T02:03:46.311 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8299999999999995e-07 2024-04-19T02:03:46.311 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.311 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.311 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:46.311 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.222723+0000", 2024-04-19T02:03:46.311 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9899999999999999e-06 2024-04-19T02:03:46.312 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.312 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.312 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:46.312 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.222735+0000", 2024-04-19T02:03:46.312 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1321999999999999e-05 2024-04-19T02:03:46.312 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.312 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.312 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:46.312 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.222743+0000", 2024-04-19T02:03:46.312 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2269999999999993e-06 2024-04-19T02:03:46.312 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.312 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.312 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:46.312 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.232017+0000", 2024-04-19T02:03:46.312 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.009274605 2024-04-19T02:03:46.313 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.313 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.313 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:46.313 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.232281+0000", 2024-04-19T02:03:46.313 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000263293 2024-04-19T02:03:46.313 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.313 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.313 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:46.313 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.234978+0000", 2024-04-19T02:03:46.313 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.002697573 2024-04-19T02:03:46.313 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.313 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.313 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:03:46.313 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.235226+0000", 2024-04-19T02:03:46.313 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00024773999999999998 2024-04-19T02:03:46.314 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:46.314 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:46.314 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:46.314 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:45.237512+0000", 2024-04-19T02:03:46.314 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0022855369999999998 2024-04-19T02:03:46.314 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.314 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:46.314 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.314 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:46.314 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:46.314 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:46.315 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T02:03:46.317 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-19T02:03:46.318 INFO:tasks.rados.rados.0.smithi059.stdout:255: writing smithi05930959-43 from 2375680 to 2998272 tid 3 2024-04-19T02:03:46.319 INFO:tasks.rados.rados.0.smithi059.stdout:255: writing smithi05930959-43 from 2998272 to 3276800 tid 4 2024-04-19T02:03:46.319 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 30 current snap is 19 2024-04-19T02:03:46.319 INFO:tasks.rados.rados.0.smithi059.stdout:256: seq_num 121 ranges {524288=491520,1015808=802816,1818624=720896,2539520=32768} 2024-04-19T02:03:46.319 INFO:tasks.rados.rados.0.smithi059.stdout:256: writing smithi05930959-30 from 524288 to 1015808 tid 1 2024-04-19T02:03:46.319 INFO:tasks.rados.rados.0.smithi059.stdout:256: writing smithi05930959-30 from 1015808 to 1818624 tid 2 2024-04-19T02:03:46.319 INFO:tasks.rados.rados.0.smithi059.stdout:256: writing smithi05930959-30 from 1818624 to 2539520 tid 3 2024-04-19T02:03:46.319 INFO:tasks.rados.rados.0.smithi059.stdout:256: writing smithi05930959-30 from 2539520 to 2572288 tid 4 2024-04-19T02:03:46.319 INFO:tasks.rados.rados.0.smithi059.stdout:257: read oid 50 snap -1 2024-04-19T02:03:46.319 INFO:tasks.rados.rados.0.smithi059.stdout:257: expect (ObjNum 29 snap 0 seq_num 29) 2024-04-19T02:03:46.319 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 17 current snap is 19 2024-04-19T02:03:46.319 INFO:tasks.rados.rados.0.smithi059.stdout:258: seq_num 122 ranges {1245184=507904,1753088=425984,2179072=442368,2621440=540672,3162112=147456} 2024-04-19T02:03:46.319 INFO:tasks.rados.rados.0.smithi059.stdout:258: writing smithi05930959-17 from 1245184 to 1753088 tid 1 2024-04-19T02:03:46.319 INFO:tasks.rados.rados.0.smithi059.stdout:258: writing smithi05930959-17 from 1753088 to 2179072 tid 2 2024-04-19T02:03:46.320 INFO:tasks.rados.rados.0.smithi059.stdout:258: writing smithi05930959-17 from 2179072 to 2621440 tid 3 2024-04-19T02:03:46.320 INFO:tasks.rados.rados.0.smithi059.stdout:258: writing smithi05930959-17 from 2621440 to 3162112 tid 4 2024-04-19T02:03:46.320 INFO:tasks.rados.rados.0.smithi059.stdout:258: writing smithi05930959-17 from 3162112 to 3309568 tid 5 2024-04-19T02:03:46.320 INFO:tasks.rados.rados.0.smithi059.stdout:259: delete oid 24 current snap is 19 2024-04-19T02:03:46.320 INFO:tasks.rados.rados.0.smithi059.stdout:254: expect (ObjNum 27 snap 0 seq_num 27) 2024-04-19T02:03:46.320 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:46.318+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:46.321 INFO:tasks.rados.rados.0.smithi059.stdout:255: finishing write tid 1 to smithi05930959-43 2024-04-19T02:03:46.321 INFO:tasks.rados.rados.0.smithi059.stdout:255: finishing write tid 2 to smithi05930959-43 2024-04-19T02:03:46.321 INFO:tasks.rados.rados.0.smithi059.stdout:255: finishing write tid 3 to smithi05930959-43 2024-04-19T02:03:46.321 INFO:tasks.rados.rados.0.smithi059.stdout:255: finishing write tid 4 to smithi05930959-43 2024-04-19T02:03:46.321 INFO:tasks.rados.rados.0.smithi059.stdout:255: finishing write tid 5 to smithi05930959-43 2024-04-19T02:03:46.321 INFO:tasks.rados.rados.0.smithi059.stdout:255: finishing write tid 6 to smithi05930959-43 2024-04-19T02:03:46.321 INFO:tasks.rados.rados.0.smithi059.stdout:255: oid 43 updating version 0 to 53 2024-04-19T02:03:46.321 INFO:tasks.rados.rados.0.smithi059.stdout:255: oid 43 updating version 53 to 56 2024-04-19T02:03:46.322 INFO:tasks.rados.rados.0.smithi059.stdout:255: oid 43 version 56 is already newer than 55 2024-04-19T02:03:46.322 INFO:tasks.rados.rados.0.smithi059.stdout:255: oid 43 version 56 is already newer than 54 2024-04-19T02:03:46.322 INFO:tasks.rados.rados.0.smithi059.stdout:255: oid 43 version 56 is already newer than 52 2024-04-19T02:03:46.322 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 43 v 56 (ObjNum 120 snap 19 seq_num 120) dirty exists 2024-04-19T02:03:46.322 INFO:tasks.rados.rados.0.smithi059.stdout:255: left oid 43 (ObjNum 120 snap 19 seq_num 120) 2024-04-19T02:03:46.322 INFO:tasks.rados.rados.0.smithi059.stdout:256: finishing write tid 1 to smithi05930959-30 2024-04-19T02:03:46.322 INFO:tasks.rados.rados.0.smithi059.stdout:256: finishing write tid 2 to smithi05930959-30 2024-04-19T02:03:46.322 INFO:tasks.rados.rados.0.smithi059.stdout:254: done (5 left) 2024-04-19T02:03:46.322 INFO:tasks.rados.rados.0.smithi059.stdout:255: done (4 left) 2024-04-19T02:03:46.322 INFO:tasks.rados.rados.0.smithi059.stdout:259: done (3 left) 2024-04-19T02:03:46.322 INFO:tasks.rados.rados.0.smithi059.stdout:260: read oid 20 snap -1 2024-04-19T02:03:46.322 INFO:tasks.rados.rados.0.smithi059.stdout:260: expect (ObjNum 94 snap 9 seq_num 94) 2024-04-19T02:03:46.322 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 25 current snap is 19 2024-04-19T02:03:46.322 INFO:tasks.rados.rados.0.smithi059.stdout:256: finishing write tid 3 to smithi05930959-30 2024-04-19T02:03:46.323 INFO:tasks.rados.rados.0.smithi059.stdout:261: seq_num 124 ranges {2998272=540672,3538944=720896,4259840=786432,5046272=49152} 2024-04-19T02:03:46.356 INFO:tasks.rados.rados.0.smithi059.stdout:261: writing smithi05930959-25 from 2998272 to 3538944 tid 1 2024-04-19T02:03:46.362 INFO:tasks.rados.rados.0.smithi059.stdout:261: writing smithi05930959-25 from 3538944 to 4259840 tid 2 2024-04-19T02:03:46.369 INFO:tasks.rados.rados.0.smithi059.stdout:261: writing smithi05930959-25 from 4259840 to 5046272 tid 3 2024-04-19T02:03:46.369 INFO:tasks.rados.rados.0.smithi059.stdout:261: writing smithi05930959-25 from 5046272 to 5095424 tid 4 2024-04-19T02:03:46.369 INFO:tasks.rados.rados.0.smithi059.stdout:256: finishing write tid 4 to smithi05930959-30 2024-04-19T02:03:46.370 INFO:tasks.rados.rados.0.smithi059.stdout:256: finishing write tid 5 to smithi05930959-30 2024-04-19T02:03:46.370 INFO:tasks.rados.rados.0.smithi059.stdout:257: expect (ObjNum 29 snap 0 seq_num 29) 2024-04-19T02:03:46.390 INFO:tasks.rados.rados.0.smithi059.stdout:256: finishing write tid 6 to smithi05930959-30 2024-04-19T02:03:46.390 INFO:tasks.rados.rados.0.smithi059.stdout:256: oid 30 updating version 0 to 40 2024-04-19T02:03:46.390 INFO:tasks.rados.rados.0.smithi059.stdout:256: oid 30 version 40 is already newer than 39 2024-04-19T02:03:46.390 INFO:tasks.rados.rados.0.smithi059.stdout:256: oid 30 version 40 is already newer than 37 2024-04-19T02:03:46.390 INFO:tasks.rados.rados.0.smithi059.stdout:256: oid 30 updating version 40 to 41 2024-04-19T02:03:46.390 INFO:tasks.rados.rados.0.smithi059.stdout:256: oid 30 version 41 is already newer than 38 2024-04-19T02:03:46.390 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 30 v 41 (ObjNum 121 snap 19 seq_num 121) dirty exists 2024-04-19T02:03:46.390 INFO:tasks.rados.rados.0.smithi059.stdout:256: left oid 30 (ObjNum 121 snap 19 seq_num 121) 2024-04-19T02:03:46.390 INFO:tasks.rados.rados.0.smithi059.stdout:258: finishing write tid 1 to smithi05930959-17 2024-04-19T02:03:46.391 INFO:tasks.rados.rados.0.smithi059.stdout:262: read oid 28 snap 17 2024-04-19T02:03:46.391 INFO:tasks.rados.rados.0.smithi059.stdout:262: expect (ObjNum 27 snap 0 seq_num 27) 2024-04-19T02:03:46.391 INFO:tasks.rados.rados.0.smithi059.stdout:256: done (5 left) 2024-04-19T02:03:46.391 INFO:tasks.rados.rados.0.smithi059.stdout:257: done (4 left) 2024-04-19T02:03:46.391 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 5 current snap is 19 2024-04-19T02:03:46.391 INFO:tasks.rados.rados.0.smithi059.stdout:263: seq_num 125 ranges {1589248=557056,2146304=475136,2621440=507904,3129344=475136,3604480=98304} 2024-04-19T02:03:46.408 INFO:tasks.rados.rados.0.smithi059.stdout:263: writing smithi05930959-5 from 1589248 to 2146304 tid 1 2024-04-19T02:03:46.413 INFO:tasks.rados.rados.0.smithi059.stdout:263: writing smithi05930959-5 from 2146304 to 2621440 tid 2 2024-04-19T02:03:46.417 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:46.417+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:46.417 INFO:tasks.rados.rados.0.smithi059.stdout:263: writing smithi05930959-5 from 2621440 to 3129344 tid 3 2024-04-19T02:03:46.421 INFO:tasks.rados.rados.0.smithi059.stdout:263: writing smithi05930959-5 from 3129344 to 3604480 tid 4 2024-04-19T02:03:46.423 INFO:tasks.rados.rados.0.smithi059.stdout:263: writing smithi05930959-5 from 3604480 to 3702784 tid 5 2024-04-19T02:03:46.423 INFO:tasks.rados.rados.0.smithi059.stdout:264: rmattr oid 2 current snap is 19 2024-04-19T02:03:46.423 INFO:tasks.rados.rados.0.smithi059.stdout:264: done (5 left) 2024-04-19T02:03:46.423 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 4 current snap is 19 2024-04-19T02:03:46.423 INFO:tasks.rados.rados.0.smithi059.stdout:265: seq_num 126 ranges {1687552=573440,2260992=622592,2883584=622592,3506176=311296} 2024-04-19T02:03:46.438 INFO:tasks.rados.rados.0.smithi059.stdout:265: writing smithi05930959-4 from 1687552 to 2260992 tid 1 2024-04-19T02:03:46.443 INFO:tasks.rados.rados.0.smithi059.stdout:265: writing smithi05930959-4 from 2260992 to 2883584 tid 2 2024-04-19T02:03:46.447 INFO:tasks.rados.rados.0.smithi059.stdout:265: writing smithi05930959-4 from 2883584 to 3506176 tid 3 2024-04-19T02:03:46.449 INFO:tasks.rados.rados.0.smithi059.stdout:265: writing smithi05930959-4 from 3506176 to 3817472 tid 4 2024-04-19T02:03:46.449 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 12 current snap is 19 2024-04-19T02:03:46.449 INFO:tasks.rados.rados.0.smithi059.stdout:266: seq_num 127 ranges {0=589824,589824=753664,1343488=475136,1818624=327680} 2024-04-19T02:03:46.453 INFO:tasks.rados.rados.0.smithi059.stdout:266: writing smithi05930959-12 from 0 to 589824 tid 1 2024-04-19T02:03:46.458 INFO:tasks.rados.rados.0.smithi059.stdout:266: writing smithi05930959-12 from 589824 to 1343488 tid 2 2024-04-19T02:03:46.461 INFO:tasks.rados.rados.0.smithi059.stdout:266: writing smithi05930959-12 from 1343488 to 1818624 tid 3 2024-04-19T02:03:46.464 INFO:tasks.rados.rados.0.smithi059.stdout:266: writing smithi05930959-12 from 1818624 to 2146304 tid 4 2024-04-19T02:03:46.464 INFO:tasks.rados.rados.0.smithi059.stdout:267: delete oid 37 current snap is 19 2024-04-19T02:03:46.464 INFO:tasks.rados.rados.0.smithi059.stdout:258: finishing write tid 2 to smithi05930959-17 2024-04-19T02:03:46.464 INFO:tasks.rados.rados.0.smithi059.stdout:258: finishing write tid 3 to smithi05930959-17 2024-04-19T02:03:46.464 INFO:tasks.rados.rados.0.smithi059.stdout:258: finishing write tid 4 to smithi05930959-17 2024-04-19T02:03:46.464 INFO:tasks.rados.rados.0.smithi059.stdout:258: finishing write tid 5 to smithi05930959-17 2024-04-19T02:03:46.464 INFO:tasks.rados.rados.0.smithi059.stdout:258: finishing write tid 6 to smithi05930959-17 2024-04-19T02:03:46.464 INFO:tasks.rados.rados.0.smithi059.stdout:258: finishing write tid 7 to smithi05930959-17 2024-04-19T02:03:46.464 INFO:tasks.rados.rados.0.smithi059.stdout:258: oid 17 updating version 0 to 123 2024-04-19T02:03:46.464 INFO:tasks.rados.rados.0.smithi059.stdout:258: oid 17 version 123 is already newer than 121 2024-04-19T02:03:46.464 INFO:tasks.rados.rados.0.smithi059.stdout:258: oid 17 updating version 123 to 124 2024-04-19T02:03:46.464 INFO:tasks.rados.rados.0.smithi059.stdout:258: oid 17 updating version 124 to 125 2024-04-19T02:03:46.465 INFO:tasks.rados.rados.0.smithi059.stdout:258: oid 17 version 125 is already newer than 120 2024-04-19T02:03:46.465 INFO:tasks.rados.rados.0.smithi059.stdout:258: oid 17 version 125 is already newer than 122 2024-04-19T02:03:46.465 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 17 v 125 (ObjNum 122 snap 19 seq_num 122) dirty exists 2024-04-19T02:03:46.465 INFO:tasks.rados.rados.0.smithi059.stdout:258: left oid 17 (ObjNum 122 snap 19 seq_num 122) 2024-04-19T02:03:46.465 INFO:tasks.rados.rados.0.smithi059.stdout:260: expect (ObjNum 94 snap 9 seq_num 94) 2024-04-19T02:03:46.465 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:46.465 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:46.465 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 0 2024-04-19T02:03:46.465 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:46.485 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-19T02:03:46.492 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-04-19T02:03:46.492 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T02:03:46.508 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-04-19T02:03:46.509 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T02:03:46.518 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:46.517+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:46.525 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-04-19T02:03:46.526 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T02:03:46.542 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-04-19T02:03:46.542 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T02:03:46.559 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-04-19T02:03:46.559 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T02:03:46.576 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-04-19T02:03:46.577 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T02:03:46.585 INFO:tasks.rados.rados.0.smithi059.stdout:261: finishing write tid 1 to smithi05930959-25 2024-04-19T02:03:46.586 INFO:tasks.rados.rados.0.smithi059.stdout:261: finishing write tid 2 to smithi05930959-25 2024-04-19T02:03:46.586 INFO:tasks.rados.rados.0.smithi059.stdout:261: finishing write tid 3 to smithi05930959-25 2024-04-19T02:03:46.586 INFO:tasks.rados.rados.0.smithi059.stdout:261: finishing write tid 4 to smithi05930959-25 2024-04-19T02:03:46.586 INFO:tasks.rados.rados.0.smithi059.stdout:261: finishing write tid 5 to smithi05930959-25 2024-04-19T02:03:46.586 INFO:tasks.rados.rados.0.smithi059.stdout:261: finishing write tid 6 to smithi05930959-25 2024-04-19T02:03:46.586 INFO:tasks.rados.rados.0.smithi059.stdout:261: oid 25 updating version 0 to 57 2024-04-19T02:03:46.586 INFO:tasks.rados.rados.0.smithi059.stdout:261: oid 25 version 57 is already newer than 56 2024-04-19T02:03:46.586 INFO:tasks.rados.rados.0.smithi059.stdout:261: oid 25 version 57 is already newer than 54 2024-04-19T02:03:46.586 INFO:tasks.rados.rados.0.smithi059.stdout:261: oid 25 version 57 is already newer than 55 2024-04-19T02:03:46.586 INFO:tasks.rados.rados.0.smithi059.stdout:261: oid 25 version 57 is already newer than 53 2024-04-19T02:03:46.586 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 25 v 57 (ObjNum 124 snap 19 seq_num 124) dirty exists 2024-04-19T02:03:46.586 INFO:tasks.rados.rados.0.smithi059.stdout:261: left oid 25 (ObjNum 124 snap 19 seq_num 124) 2024-04-19T02:03:46.586 INFO:tasks.rados.rados.0.smithi059.stdout:262: expect (ObjNum 27 snap 0 seq_num 27) 2024-04-19T02:03:46.594 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-04-19T02:03:46.594 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T02:03:46.598 INFO:tasks.rados.rados.0.smithi059.stdout:263: finishing write tid 1 to smithi05930959-5 2024-04-19T02:03:46.598 INFO:tasks.rados.rados.0.smithi059.stdout:263: finishing write tid 2 to smithi05930959-5 2024-04-19T02:03:46.598 INFO:tasks.rados.rados.0.smithi059.stdout:263: finishing write tid 3 to smithi05930959-5 2024-04-19T02:03:46.598 INFO:tasks.rados.rados.0.smithi059.stdout:263: finishing write tid 4 to smithi05930959-5 2024-04-19T02:03:46.598 INFO:tasks.rados.rados.0.smithi059.stdout:258: done (7 left) 2024-04-19T02:03:46.598 INFO:tasks.rados.rados.0.smithi059.stdout:260: done (6 left) 2024-04-19T02:03:46.598 INFO:tasks.rados.rados.0.smithi059.stdout:261: done (5 left) 2024-04-19T02:03:46.598 INFO:tasks.rados.rados.0.smithi059.stdout:262: done (4 left) 2024-04-19T02:03:46.598 INFO:tasks.rados.rados.0.smithi059.stdout:267: done (3 left) 2024-04-19T02:03:46.598 INFO:tasks.rados.rados.0.smithi059.stdout:268: snap_remove snap 18 2024-04-19T02:03:46.613 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-04-19T02:03:46.613 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T02:03:46.619 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:46.622+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:46.634 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-04-19T02:03:46.634 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T02:03:46.653 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:46.653 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:46.655 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-04-19T02:03:46.655 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T02:03:46.667 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-19T02:03:46.676 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-04-19T02:03:46.677 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T02:03:46.699 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-04-19T02:03:46.699 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T02:03:46.720 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:46.721+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:46.790 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:46.790 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:46.804 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-19T02:03:46.820 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:46.817+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:46.921 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:46.922+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:46.972 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:46.972 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:46.986 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-19T02:03:47.031 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:47.033+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:47.052 INFO:tasks.rados.rados.0.smithi059.stdout:268: done (3 left) 2024-04-19T02:03:47.052 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 13 current snap is 19 2024-04-19T02:03:47.052 INFO:tasks.rados.rados.0.smithi059.stdout:269: seq_num 129 ranges {1163264=622592,1785856=655360,2441216=688128,3129344=212992} 2024-04-19T02:03:47.071 INFO:tasks.rados.rados.0.smithi059.stdout:269: writing smithi05930959-13 from 1163264 to 1785856 tid 1 2024-04-19T02:03:47.080 INFO:tasks.rados.rados.0.smithi059.stdout:269: writing smithi05930959-13 from 1785856 to 2441216 tid 2 2024-04-19T02:03:47.087 INFO:tasks.rados.rados.0.smithi059.stdout:269: writing smithi05930959-13 from 2441216 to 3129344 tid 3 2024-04-19T02:03:47.090 INFO:tasks.rados.rados.0.smithi059.stdout:269: writing smithi05930959-13 from 3129344 to 3342336 tid 4 2024-04-19T02:03:47.090 INFO:tasks.rados.rados.0.smithi059.stdout:270: delete oid 43 current snap is 19 2024-04-19T02:03:47.090 INFO:tasks.rados.rados.0.smithi059.stdout:263: finishing write tid 5 to smithi05930959-5 2024-04-19T02:03:47.090 INFO:tasks.rados.rados.0.smithi059.stdout:263: finishing write tid 6 to smithi05930959-5 2024-04-19T02:03:47.090 INFO:tasks.rados.rados.0.smithi059.stdout:263: finishing write tid 7 to smithi05930959-5 2024-04-19T02:03:47.090 INFO:tasks.rados.rados.0.smithi059.stdout:263: oid 5 updating version 0 to 59 2024-04-19T02:03:47.091 INFO:tasks.rados.rados.0.smithi059.stdout:263: oid 5 version 59 is already newer than 57 2024-04-19T02:03:47.091 INFO:tasks.rados.rados.0.smithi059.stdout:263: oid 5 updating version 59 to 60 2024-04-19T02:03:47.091 INFO:tasks.rados.rados.0.smithi059.stdout:263: oid 5 updating version 60 to 62 2024-04-19T02:03:47.091 INFO:tasks.rados.rados.0.smithi059.stdout:263: oid 5 version 62 is already newer than 58 2024-04-19T02:03:47.091 INFO:tasks.rados.rados.0.smithi059.stdout:263: oid 5 version 62 is already newer than 61 2024-04-19T02:03:47.091 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 5 v 62 (ObjNum 125 snap 19 seq_num 125) dirty exists 2024-04-19T02:03:47.091 INFO:tasks.rados.rados.0.smithi059.stdout:263: left oid 5 (ObjNum 125 snap 19 seq_num 125) 2024-04-19T02:03:47.091 INFO:tasks.rados.rados.0.smithi059.stdout:265: finishing write tid 1 to smithi05930959-4 2024-04-19T02:03:47.091 INFO:tasks.rados.rados.0.smithi059.stdout:266: finishing write tid 1 to smithi05930959-12 2024-04-19T02:03:47.091 INFO:tasks.rados.rados.0.smithi059.stdout:265: finishing write tid 2 to smithi05930959-4 2024-04-19T02:03:47.091 INFO:tasks.rados.rados.0.smithi059.stdout:265: finishing write tid 3 to smithi05930959-4 2024-04-19T02:03:47.091 INFO:tasks.rados.rados.0.smithi059.stdout:265: finishing write tid 4 to smithi05930959-4 2024-04-19T02:03:47.091 INFO:tasks.rados.rados.0.smithi059.stdout:266: finishing write tid 2 to smithi05930959-12 2024-04-19T02:03:47.091 INFO:tasks.rados.rados.0.smithi059.stdout:265: finishing write tid 5 to smithi05930959-4 2024-04-19T02:03:47.092 INFO:tasks.rados.rados.0.smithi059.stdout:266: finishing write tid 3 to smithi05930959-12 2024-04-19T02:03:47.092 INFO:tasks.rados.rados.0.smithi059.stdout:266: finishing write tid 4 to smithi05930959-12 2024-04-19T02:03:47.092 INFO:tasks.rados.rados.0.smithi059.stdout:266: finishing write tid 5 to smithi05930959-12 2024-04-19T02:03:47.092 INFO:tasks.rados.rados.0.smithi059.stdout:265: finishing write tid 6 to smithi05930959-4 2024-04-19T02:03:47.092 INFO:tasks.rados.rados.0.smithi059.stdout:265: oid 4 updating version 0 to 70 2024-04-19T02:03:47.092 INFO:tasks.rados.rados.0.smithi059.stdout:265: oid 4 version 70 is already newer than 68 2024-04-19T02:03:47.092 INFO:tasks.rados.rados.0.smithi059.stdout:265: oid 4 version 70 is already newer than 67 2024-04-19T02:03:47.092 INFO:tasks.rados.rados.0.smithi059.stdout:265: oid 4 version 70 is already newer than 69 2024-04-19T02:03:47.092 INFO:tasks.rados.rados.0.smithi059.stdout:265: oid 4 updating version 70 to 71 2024-04-19T02:03:47.092 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 4 v 71 (ObjNum 126 snap 19 seq_num 126) dirty exists 2024-04-19T02:03:47.092 INFO:tasks.rados.rados.0.smithi059.stdout:265: left oid 4 (ObjNum 126 snap 19 seq_num 126) 2024-04-19T02:03:47.092 INFO:tasks.rados.rados.0.smithi059.stdout:266: finishing write tid 6 to smithi05930959-12 2024-04-19T02:03:47.092 INFO:tasks.rados.rados.0.smithi059.stdout:266: oid 12 updating version 0 to 45 2024-04-19T02:03:47.093 INFO:tasks.rados.rados.0.smithi059.stdout:266: oid 12 version 45 is already newer than 44 2024-04-19T02:03:47.093 INFO:tasks.rados.rados.0.smithi059.stdout:266: oid 12 updating version 45 to 46 2024-04-19T02:03:47.093 INFO:tasks.rados.rados.0.smithi059.stdout:266: oid 12 version 46 is already newer than 42 2024-04-19T02:03:47.093 INFO:tasks.rados.rados.0.smithi059.stdout:266: oid 12 version 46 is already newer than 43 2024-04-19T02:03:47.093 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 12 v 46 (ObjNum 127 snap 19 seq_num 127) dirty exists 2024-04-19T02:03:47.093 INFO:tasks.rados.rados.0.smithi059.stdout:266: left oid 12 (ObjNum 127 snap 19 seq_num 127) 2024-04-19T02:03:47.093 INFO:tasks.rados.rados.0.smithi059.stdout:269: finishing write tid 1 to smithi05930959-13 2024-04-19T02:03:47.093 INFO:tasks.rados.rados.0.smithi059.stdout:269: finishing write tid 2 to smithi05930959-13 2024-04-19T02:03:47.096 INFO:tasks.rados.rados.0.smithi059.stdout:263: done (4 left) 2024-04-19T02:03:47.096 INFO:tasks.rados.rados.0.smithi059.stdout:265: done (3 left) 2024-04-19T02:03:47.096 INFO:tasks.rados.rados.0.smithi059.stdout:266: done (2 left) 2024-04-19T02:03:47.096 INFO:tasks.rados.rados.0.smithi059.stdout:270: done (1 left) 2024-04-19T02:03:47.096 INFO:tasks.rados.rados.0.smithi059.stdout:271: delete oid 25 current snap is 19 2024-04-19T02:03:47.096 INFO:tasks.rados.rados.0.smithi059.stdout:269: finishing write tid 3 to smithi05930959-13 2024-04-19T02:03:47.097 INFO:tasks.rados.rados.0.smithi059.stdout:269: finishing write tid 4 to smithi05930959-13 2024-04-19T02:03:47.098 INFO:tasks.rados.rados.0.smithi059.stdout:269: finishing write tid 5 to smithi05930959-13 2024-04-19T02:03:47.101 INFO:tasks.rados.rados.0.smithi059.stdout:269: finishing write tid 6 to smithi05930959-13 2024-04-19T02:03:47.101 INFO:tasks.rados.rados.0.smithi059.stdout:269: oid 13 updating version 0 to 40 2024-04-19T02:03:47.101 INFO:tasks.rados.rados.0.smithi059.stdout:269: oid 13 version 40 is already newer than 37 2024-04-19T02:03:47.101 INFO:tasks.rados.rados.0.smithi059.stdout:269: oid 13 version 40 is already newer than 38 2024-04-19T02:03:47.101 INFO:tasks.rados.rados.0.smithi059.stdout:269: oid 13 version 40 is already newer than 36 2024-04-19T02:03:47.101 INFO:tasks.rados.rados.0.smithi059.stdout:269: oid 13 version 40 is already newer than 39 2024-04-19T02:03:47.101 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 13 v 40 (ObjNum 129 snap 19 seq_num 129) dirty exists 2024-04-19T02:03:47.101 INFO:tasks.rados.rados.0.smithi059.stdout:269: left oid 13 (ObjNum 129 snap 19 seq_num 129) 2024-04-19T02:03:47.102 INFO:tasks.rados.rados.0.smithi059.stdout:269: done (1 left) 2024-04-19T02:03:47.102 INFO:tasks.rados.rados.0.smithi059.stdout:271: done (0 left) 2024-04-19T02:03:47.102 INFO:tasks.rados.rados.0.smithi059.stdout:272: delete oid 8 current snap is 19 2024-04-19T02:03:47.107 INFO:tasks.rados.rados.0.smithi059.stdout:272: done (0 left) 2024-04-19T02:03:47.107 INFO:tasks.rados.rados.0.smithi059.stdout:273: copy_from oid 21 from oid 44 current snap is 19 2024-04-19T02:03:47.108 INFO:tasks.rados.rados.0.smithi059.stdout:274: snap_remove snap 17 2024-04-19T02:03:47.122 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:47.122 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:47.123 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:47.121+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:47.136 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-19T02:03:47.197 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-19T02:03:47.223 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:03:47.225+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:03:47.302 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:47.302 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:47.316 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-19T02:03:47.359 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:03:47.361+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:03:47.425 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:47.425+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:47.484 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:47.484 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:47.498 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-19T02:03:47.526 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:47.526+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:47.622 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:47.622 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:47.627 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:03:47.629+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:03:47.636 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-19T02:03:47.728 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:47.730+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:47.806 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:47.806 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:47.818 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-19T02:03:47.828 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:47.830+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:47.930 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:47.930+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:47.949 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:47.949 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:47.963 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-19T02:03:48.030 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:48.029+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:48.046 INFO:teuthology.orchestra.run.smithi059.stderr:nodeep-scrub is set 2024-04-19T02:03:48.072 INFO:tasks.rados.rados.0.smithi059.stdout:274: done (1 left) 2024-04-19T02:03:48.073 INFO:tasks.rados.rados.0.smithi059.stdout:275: rollback oid 22 current snap is 19 2024-04-19T02:03:48.073 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 22 to 16 2024-04-19T02:03:48.073 INFO:tasks.rados.rados.0.smithi059.stdout:276: snap_create 2024-04-19T02:03:48.073 INFO:tasks.rados.rados.0.smithi059.stdout:273: finishing copy_from to smithi05930959-21 2024-04-19T02:03:48.073 INFO:tasks.rados.rados.0.smithi059.stdout:273: got expected ENOENT (src dne) 2024-04-19T02:03:48.073 INFO:tasks.rados.rados.0.smithi059.stdout:273: finishing copy_from racing read to smithi05930959-21 2024-04-19T02:03:48.088 INFO:tasks.rados.rados.0.smithi059.stdout:275: finishing rollback tid 0 to smithi05930959-22 2024-04-19T02:03:48.090 INFO:tasks.rados.rados.0.smithi059.stdout:275: finishing rollback tid 1 to smithi05930959-22 2024-04-19T02:03:48.090 INFO:tasks.rados.rados.0.smithi059.stdout:275: finishing rollback tid 2 to smithi05930959-22 2024-04-19T02:03:48.090 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 22 v 131 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-19T02:03:48.101 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:48.101 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:48.115 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-19T02:03:48.131 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:48.133+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:48.232 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:48.234+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:48.279 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:48.280 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:48.293 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-19T02:03:48.333 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:03:48.333+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:03:48.415 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:48.415 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:48.429 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-19T02:03:48.434 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:48.434+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:48.534 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:48.533+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:48.550 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:48.550 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:48.564 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-19T02:03:48.650 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:48.649+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:48.720 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T02:03:48.730 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:48.730 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:48.742 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-19T02:03:48.774 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:48.774+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:48.840 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:48.841+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:48.878 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-04-19T02:03:48.879 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T02:03:48.895 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-04-19T02:03:48.895 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T02:03:48.911 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:48.912 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:48.912 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-04-19T02:03:48.912 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T02:03:48.926 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-19T02:03:48.930 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-04-19T02:03:48.930 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T02:03:48.942 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:48.946+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:48.950 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-04-19T02:03:48.950 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T02:03:48.971 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-04-19T02:03:48.972 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T02:03:48.992 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-04-19T02:03:48.992 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T02:03:49.014 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-04-19T02:03:49.014 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T02:03:49.037 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-04-19T02:03:49.037 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T02:03:49.041 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:49.041+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:49.044 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:49.044 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:49.053 INFO:tasks.rados.rados.0.smithi059.stdout:273: done (2 left) 2024-04-19T02:03:49.053 INFO:tasks.rados.rados.0.smithi059.stdout:275: done (1 left) 2024-04-19T02:03:49.053 INFO:tasks.rados.rados.0.smithi059.stdout:276: done (0 left) 2024-04-19T02:03:49.053 INFO:tasks.rados.rados.0.smithi059.stdout:277: snap_create 2024-04-19T02:03:49.060 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-19T02:03:49.061 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-04-19T02:03:49.061 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T02:03:49.082 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-04-19T02:03:49.083 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T02:03:49.102 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-04-19T02:03:49.103 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T02:03:49.163 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:49.166+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:49.186 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:49.186 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-04-19T02:03:49.186 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-04-19T02:03:49.186 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-04-19T02:03:49.186 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.186 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:293 3.5s0 3:a41c791e:::smithi05930959-15:head [delete] snapc 6=[6,5] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e37)", 2024-04-19T02:03:49.186 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:21.689881+0000", 2024-04-19T02:03:49.186 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 27.496665011000001, 2024-04-19T02:03:49.186 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011022188, 2024-04-19T02:03:49.186 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.187 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.187 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:49.187 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:49.187 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:49.187 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:49.187 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 293 2024-04-19T02:03:49.187 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.187 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.187 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.187 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.187 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.689881+0000", 2024-04-19T02:03:49.187 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.187 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.187 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.187 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.187 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.689881+0000", 2024-04-19T02:03:49.188 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.188 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.188 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.188 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.188 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.689883+0000", 2024-04-19T02:03:49.188 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6170000000000001e-06 2024-04-19T02:03:49.188 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.188 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.188 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.188 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.689896+0000", 2024-04-19T02:03:49.188 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2826000000000001e-05 2024-04-19T02:03:49.188 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.188 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.188 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.188 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.689930+0000", 2024-04-19T02:03:49.188 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4289999999999999e-05 2024-04-19T02:03:49.189 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.189 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.189 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.189 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.689949+0000", 2024-04-19T02:03:49.189 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8921000000000001e-05 2024-04-19T02:03:49.189 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.189 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.189 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.189 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.690106+0000", 2024-04-19T02:03:49.189 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000156496 2024-04-19T02:03:49.189 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.189 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.189 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op must wait for map", 2024-04-19T02:03:49.189 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.690123+0000", 2024-04-19T02:03:49.189 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6949e-05 2024-04-19T02:03:49.189 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.190 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.190 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.190 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.693813+0000", 2024-04-19T02:03:49.190 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0036899419999999999 2024-04-19T02:03:49.190 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.190 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.190 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:49.190 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.693962+0000", 2024-04-19T02:03:49.190 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014913 2024-04-19T02:03:49.190 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.190 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.190 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:49.190 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.695407+0000", 2024-04-19T02:03:49.190 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001444823 2024-04-19T02:03:49.190 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.191 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.191 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:49.191 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.698101+0000", 2024-04-19T02:03:49.191 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002694288 2024-04-19T02:03:49.191 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.191 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.191 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:49.191 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.700773+0000", 2024-04-19T02:03:49.191 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0026718940000000002 2024-04-19T02:03:49.191 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.191 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.191 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.191 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:21.700903+0000", 2024-04-19T02:03:49.191 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013001199999999999 2024-04-19T02:03:49.191 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.191 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.191 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.192 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.192 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.192 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.4s2 42/26 ECSubWrite(tid=44, reqid=osd.1.0:44, at_version=42'32, trim_to=0'0, roll_forward_to=40'28))", 2024-04-19T02:03:49.192 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.739493+0000", 2024-04-19T02:03:49.192 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 22.447052456000002, 2024-04-19T02:03:49.192 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015165089, 2024-04-19T02:03:49.192 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.192 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.192 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:49.192 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.192 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.192 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.192 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.739493+0000", 2024-04-19T02:03:49.192 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.192 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.193 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.193 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.193 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.739493+0000", 2024-04-19T02:03:49.193 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.193 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.193 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.193 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.193 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.739495+0000", 2024-04-19T02:03:49.193 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8729999999999999e-06 2024-04-19T02:03:49.193 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.193 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.193 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.193 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.739550+0000", 2024-04-19T02:03:49.193 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5080000000000001e-05 2024-04-19T02:03:49.193 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.193 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.194 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.194 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.739584+0000", 2024-04-19T02:03:49.194 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3396999999999999e-05 2024-04-19T02:03:49.194 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.194 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.194 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.194 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.739611+0000", 2024-04-19T02:03:49.194 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7733999999999999e-05 2024-04-19T02:03:49.194 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.194 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.194 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.194 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.740435+0000", 2024-04-19T02:03:49.194 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00082331600000000004 2024-04-19T02:03:49.194 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.194 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.194 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:49.195 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.740463+0000", 2024-04-19T02:03:49.195 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7991999999999999e-05 2024-04-19T02:03:49.195 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.195 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.195 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:49.195 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.754535+0000", 2024-04-19T02:03:49.195 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014072092 2024-04-19T02:03:49.195 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.195 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.195 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.195 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.754658+0000", 2024-04-19T02:03:49.195 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012360499999999999 2024-04-19T02:03:49.195 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.195 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.195 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.195 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.195 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.195 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.8s7 42/26 ECSubWrite(tid=71, reqid=osd.4.0:71, at_version=42'33, trim_to=0'0, roll_forward_to=37'31))", 2024-04-19T02:03:49.196 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.747600+0000", 2024-04-19T02:03:49.196 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 22.438945353000001, 2024-04-19T02:03:49.196 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012073659, 2024-04-19T02:03:49.196 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.196 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.196 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:49.196 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.196 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.196 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.196 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747600+0000", 2024-04-19T02:03:49.196 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.196 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.196 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.196 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.196 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747600+0000", 2024-04-19T02:03:49.196 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.196 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.197 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.197 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.197 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747601+0000", 2024-04-19T02:03:49.197 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5400000000000001e-07 2024-04-19T02:03:49.197 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.197 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.197 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.197 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747647+0000", 2024-04-19T02:03:49.197 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6529000000000001e-05 2024-04-19T02:03:49.197 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.197 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.197 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.197 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747671+0000", 2024-04-19T02:03:49.197 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3425e-05 2024-04-19T02:03:49.197 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.197 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.197 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.198 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747702+0000", 2024-04-19T02:03:49.198 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1531e-05 2024-04-19T02:03:49.198 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.198 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.198 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.198 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749865+0000", 2024-04-19T02:03:49.198 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0021629150000000001 2024-04-19T02:03:49.198 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.198 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.198 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:49.198 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749891+0000", 2024-04-19T02:03:49.198 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5568e-05 2024-04-19T02:03:49.198 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.198 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.198 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:49.198 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759578+0000", 2024-04-19T02:03:49.198 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0096872139999999995 2024-04-19T02:03:49.199 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.199 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.199 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.199 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759674+0000", 2024-04-19T02:03:49.199 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5922999999999996e-05 2024-04-19T02:03:49.199 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.199 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.199 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.199 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.199 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.199 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.bs5 42/26 ECSubWrite(tid=133, reqid=osd.3.0:133, at_version=42'21, trim_to=0'0, roll_forward_to=39'19))", 2024-04-19T02:03:49.199 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.748033+0000", 2024-04-19T02:03:49.199 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 22.438512892999999, 2024-04-19T02:03:49.199 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012077704999999999, 2024-04-19T02:03:49.199 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.199 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.199 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:49.200 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.200 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.200 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.200 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748033+0000", 2024-04-19T02:03:49.200 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.200 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.200 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.200 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.200 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748033+0000", 2024-04-19T02:03:49.200 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.200 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.200 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.200 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.200 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748035+0000", 2024-04-19T02:03:49.200 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.75e-06 2024-04-19T02:03:49.200 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.200 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.201 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.201 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748087+0000", 2024-04-19T02:03:49.201 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2456000000000001e-05 2024-04-19T02:03:49.201 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.201 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.201 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.201 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748118+0000", 2024-04-19T02:03:49.201 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0728000000000001e-05 2024-04-19T02:03:49.201 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.201 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.201 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.201 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748141+0000", 2024-04-19T02:03:49.201 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2824e-05 2024-04-19T02:03:49.201 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.201 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.201 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.201 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748342+0000", 2024-04-19T02:03:49.202 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000201444 2024-04-19T02:03:49.202 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.202 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.202 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:49.202 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748366+0000", 2024-04-19T02:03:49.202 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4278999999999999e-05 2024-04-19T02:03:49.202 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.202 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.202 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:49.202 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760028+0000", 2024-04-19T02:03:49.202 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011662077 2024-04-19T02:03:49.202 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.202 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.202 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.202 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760111+0000", 2024-04-19T02:03:49.203 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2146999999999998e-05 2024-04-19T02:03:49.203 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.203 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.203 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.203 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.203 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.203 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.bs5 42/26 ECSubWrite(tid=135, reqid=osd.3.0:135, at_version=42'23, trim_to=0'0, roll_forward_to=39'19))", 2024-04-19T02:03:49.203 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.749533+0000", 2024-04-19T02:03:49.203 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 22.437012342999999, 2024-04-19T02:03:49.203 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012068692000000001, 2024-04-19T02:03:49.203 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.203 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.203 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:49.203 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.203 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.203 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.203 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749533+0000", 2024-04-19T02:03:49.203 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.204 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.204 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.204 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.204 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749533+0000", 2024-04-19T02:03:49.204 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.204 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.204 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.204 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.204 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749535+0000", 2024-04-19T02:03:49.204 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7999999999999999e-06 2024-04-19T02:03:49.204 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.204 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.204 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.204 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749586+0000", 2024-04-19T02:03:49.204 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1202999999999997e-05 2024-04-19T02:03:49.204 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.205 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.205 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.205 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749618+0000", 2024-04-19T02:03:49.205 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1393999999999998e-05 2024-04-19T02:03:49.205 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.205 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.205 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.205 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749817+0000", 2024-04-19T02:03:49.205 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00019913799999999999 2024-04-19T02:03:49.205 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.205 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.205 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.205 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.757176+0000", 2024-04-19T02:03:49.205 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0073591949999999998 2024-04-19T02:03:49.205 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.205 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.205 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:49.206 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.757200+0000", 2024-04-19T02:03:49.206 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4388000000000001e-05 2024-04-19T02:03:49.206 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.206 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.206 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:49.206 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761527+0000", 2024-04-19T02:03:49.206 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0043261899999999997 2024-04-19T02:03:49.206 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.206 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.206 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.206 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761602+0000", 2024-04-19T02:03:49.206 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5383999999999995e-05 2024-04-19T02:03:49.206 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.206 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.206 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.206 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.207 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.207 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.1s4 47/26 ECSubWrite(tid=29, reqid=osd.0.0:29, at_version=47'15, trim_to=0'0, roll_forward_to=46'13))", 2024-04-19T02:03:49.207 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.087131+0000", 2024-04-19T02:03:49.207 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 18.099414562, 2024-04-19T02:03:49.207 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.028478961000000001, 2024-04-19T02:03:49.207 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.207 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.207 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:49.207 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.207 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.207 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.207 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087131+0000", 2024-04-19T02:03:49.207 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.207 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.207 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.207 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.207 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087131+0000", 2024-04-19T02:03:49.208 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.208 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.208 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.208 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.208 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087132+0000", 2024-04-19T02:03:49.208 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4000000000000001e-07 2024-04-19T02:03:49.208 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.208 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.208 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.208 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087185+0000", 2024-04-19T02:03:49.208 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3016999999999997e-05 2024-04-19T02:03:49.208 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.208 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.208 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.208 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087232+0000", 2024-04-19T02:03:49.208 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6989e-05 2024-04-19T02:03:49.208 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.208 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.209 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.209 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087259+0000", 2024-04-19T02:03:49.209 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7199000000000002e-05 2024-04-19T02:03:49.209 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.209 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.209 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.209 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087431+0000", 2024-04-19T02:03:49.209 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017193199999999999 2024-04-19T02:03:49.209 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.209 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.209 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:49.209 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087457+0000", 2024-04-19T02:03:49.209 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6148000000000001e-05 2024-04-19T02:03:49.209 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.209 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.209 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:49.210 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.115506+0000", 2024-04-19T02:03:49.210 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.028048812999999999 2024-04-19T02:03:49.210 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.210 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.210 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.210 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.115610+0000", 2024-04-19T02:03:49.210 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010412300000000001 2024-04-19T02:03:49.210 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.210 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.210 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.210 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.210 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.210 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.1s4 47/26 ECSubWrite(tid=30, reqid=osd.0.0:30, at_version=47'17, trim_to=0'0, roll_forward_to=46'13))", 2024-04-19T02:03:49.210 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.087466+0000", 2024-04-19T02:03:49.210 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 18.099080175000001, 2024-04-19T02:03:49.210 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.030894781, 2024-04-19T02:03:49.211 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.211 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.211 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:49.211 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.211 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.211 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.211 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087466+0000", 2024-04-19T02:03:49.211 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.211 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.211 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.211 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.211 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087466+0000", 2024-04-19T02:03:49.211 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.211 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.211 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.211 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.212 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087466+0000", 2024-04-19T02:03:49.212 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0400000000000005e-07 2024-04-19T02:03:49.212 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.212 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.212 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.212 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087524+0000", 2024-04-19T02:03:49.212 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7632e-05 2024-04-19T02:03:49.212 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.212 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.212 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.212 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087550+0000", 2024-04-19T02:03:49.212 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5871999999999999e-05 2024-04-19T02:03:49.212 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.212 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.212 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.212 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087570+0000", 2024-04-19T02:03:49.213 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0531999999999999e-05 2024-04-19T02:03:49.213 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.213 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.213 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.213 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.091486+0000", 2024-04-19T02:03:49.213 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039151009999999998 2024-04-19T02:03:49.213 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.213 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.213 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:49.213 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.091519+0000", 2024-04-19T02:03:49.213 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3874999999999997e-05 2024-04-19T02:03:49.213 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.213 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.213 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:49.213 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.118273+0000", 2024-04-19T02:03:49.214 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.026753835 2024-04-19T02:03:49.214 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.214 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.214 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.214 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.118360+0000", 2024-04-19T02:03:49.214 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7029999999999999e-05 2024-04-19T02:03:49.214 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.214 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.214 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.214 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.214 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.214 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s2 47/26 ECSubWrite(tid=181, reqid=osd.3.0:181, at_version=47'87, trim_to=0'0, roll_forward_to=46'85))", 2024-04-19T02:03:49.214 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.088687+0000", 2024-04-19T02:03:49.214 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 18.097858791, 2024-04-19T02:03:49.214 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.027025486000000001, 2024-04-19T02:03:49.214 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.215 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.215 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:49.215 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.215 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.215 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.215 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088687+0000", 2024-04-19T02:03:49.215 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.215 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.215 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.215 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.215 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088687+0000", 2024-04-19T02:03:49.215 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.215 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.215 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.215 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.215 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088687+0000", 2024-04-19T02:03:49.216 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7999999999999996e-07 2024-04-19T02:03:49.216 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.216 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.216 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.216 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088786+0000", 2024-04-19T02:03:49.216 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8510000000000004e-05 2024-04-19T02:03:49.216 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.216 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.216 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.216 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088822+0000", 2024-04-19T02:03:49.216 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6284e-05 2024-04-19T02:03:49.216 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.216 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.216 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.216 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088846+0000", 2024-04-19T02:03:49.216 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4125999999999999e-05 2024-04-19T02:03:49.216 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.217 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.217 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.217 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089107+0000", 2024-04-19T02:03:49.217 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00026044799999999998 2024-04-19T02:03:49.217 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.217 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.217 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:49.217 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089134+0000", 2024-04-19T02:03:49.217 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7067999999999999e-05 2024-04-19T02:03:49.217 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.217 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.217 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:49.217 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.115620+0000", 2024-04-19T02:03:49.217 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.026486464000000001 2024-04-19T02:03:49.217 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.217 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.218 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.218 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.115712+0000", 2024-04-19T02:03:49.218 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2106000000000002e-05 2024-04-19T02:03:49.218 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.218 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.218 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.218 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.218 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.218 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.bs5 47/26 ECSubWrite(tid=180, reqid=osd.3.0:180, at_version=47'36, trim_to=0'0, roll_forward_to=46'34))", 2024-04-19T02:03:49.218 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.088907+0000", 2024-04-19T02:03:49.218 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 18.097639182000002, 2024-04-19T02:03:49.218 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.028441793999999999, 2024-04-19T02:03:49.218 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.218 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.219 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:49.219 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.219 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.219 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.219 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088907+0000", 2024-04-19T02:03:49.219 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.219 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.219 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.219 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.219 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088907+0000", 2024-04-19T02:03:49.219 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.219 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.219 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.219 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.219 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088908+0000", 2024-04-19T02:03:49.219 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0100000000000001e-06 2024-04-19T02:03:49.219 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.219 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.220 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.220 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088952+0000", 2024-04-19T02:03:49.220 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4613000000000001e-05 2024-04-19T02:03:49.220 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.220 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.220 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.220 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088975+0000", 2024-04-19T02:03:49.220 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2518999999999999e-05 2024-04-19T02:03:49.220 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.220 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.220 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.220 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088993+0000", 2024-04-19T02:03:49.220 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7855000000000001e-05 2024-04-19T02:03:49.220 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.220 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.220 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.220 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089130+0000", 2024-04-19T02:03:49.221 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013709099999999999 2024-04-19T02:03:49.221 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.221 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.221 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:49.221 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089155+0000", 2024-04-19T02:03:49.221 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4924999999999999e-05 2024-04-19T02:03:49.221 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.221 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.221 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:49.221 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.117263+0000", 2024-04-19T02:03:49.221 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.028107953000000001 2024-04-19T02:03:49.221 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.221 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.221 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.221 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.117348+0000", 2024-04-19T02:03:49.221 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5828000000000006e-05 2024-04-19T02:03:49.221 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.222 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.222 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.222 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.222 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.222 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s2 47/26 ECSubWrite(tid=182, reqid=osd.3.0:182, at_version=47'89, trim_to=0'0, roll_forward_to=46'85))", 2024-04-19T02:03:49.222 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.089036+0000", 2024-04-19T02:03:49.222 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 18.097510153999998, 2024-04-19T02:03:49.222 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.027692491, 2024-04-19T02:03:49.222 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.222 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.222 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:49.222 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.222 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.222 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.223 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089036+0000", 2024-04-19T02:03:49.223 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.223 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.223 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.223 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.223 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089036+0000", 2024-04-19T02:03:49.223 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.223 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.223 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.223 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.223 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089036+0000", 2024-04-19T02:03:49.223 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7599999999999996e-07 2024-04-19T02:03:49.223 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.223 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.223 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.223 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089097+0000", 2024-04-19T02:03:49.223 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0223000000000003e-05 2024-04-19T02:03:49.224 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.224 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.224 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.224 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089117+0000", 2024-04-19T02:03:49.224 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0724999999999999e-05 2024-04-19T02:03:49.224 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.224 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.224 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.224 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089134+0000", 2024-04-19T02:03:49.224 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7065e-05 2024-04-19T02:03:49.224 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.224 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.224 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.224 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.091684+0000", 2024-04-19T02:03:49.224 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0025499780000000001 2024-04-19T02:03:49.224 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.225 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.225 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:49.225 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.091723+0000", 2024-04-19T02:03:49.225 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8787e-05 2024-04-19T02:03:49.225 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.225 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.225 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:49.225 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.116611+0000", 2024-04-19T02:03:49.225 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.024888321000000001 2024-04-19T02:03:49.225 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.225 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.225 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.225 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.116728+0000", 2024-04-19T02:03:49.225 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000116616 2024-04-19T02:03:49.225 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.225 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.225 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.225 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.226 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.226 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.bs5 47/26 ECSubWrite(tid=183, reqid=osd.3.0:183, at_version=47'38, trim_to=0'0, roll_forward_to=46'34))", 2024-04-19T02:03:49.226 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.089162+0000", 2024-04-19T02:03:49.226 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 18.097383786999998, 2024-04-19T02:03:49.226 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.031704567000000003, 2024-04-19T02:03:49.226 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.226 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.226 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:49.226 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.226 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.226 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.226 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089162+0000", 2024-04-19T02:03:49.226 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.226 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.227 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.227 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.227 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089162+0000", 2024-04-19T02:03:49.227 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.227 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.227 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.227 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.227 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089163+0000", 2024-04-19T02:03:49.227 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8999999999999996e-07 2024-04-19T02:03:49.227 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.227 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.227 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.227 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089208+0000", 2024-04-19T02:03:49.227 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5670999999999997e-05 2024-04-19T02:03:49.227 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.227 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.227 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.228 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089231+0000", 2024-04-19T02:03:49.228 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2898999999999999e-05 2024-04-19T02:03:49.228 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.228 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.228 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.228 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089248+0000", 2024-04-19T02:03:49.228 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7170999999999999e-05 2024-04-19T02:03:49.228 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.228 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.228 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.228 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.091699+0000", 2024-04-19T02:03:49.228 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0024503010000000002 2024-04-19T02:03:49.228 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.228 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.228 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:49.228 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.091731+0000", 2024-04-19T02:03:49.228 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2301000000000001e-05 2024-04-19T02:03:49.229 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.229 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.229 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:49.229 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.120784+0000", 2024-04-19T02:03:49.229 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.029052923000000001 2024-04-19T02:03:49.229 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.229 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.229 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.229 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.120866+0000", 2024-04-19T02:03:49.229 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2610999999999999e-05 2024-04-19T02:03:49.229 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.229 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.229 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.229 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.229 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.229 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.8s7 47/26 ECSubWrite(tid=86, reqid=osd.4.0:86, at_version=47'44, trim_to=0'0, roll_forward_to=46'42))", 2024-04-19T02:03:49.229 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.092233+0000", 2024-04-19T02:03:49.230 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 18.094313171, 2024-04-19T02:03:49.230 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.026209364999999998, 2024-04-19T02:03:49.230 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.230 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.230 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:49.230 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.230 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.230 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.230 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092233+0000", 2024-04-19T02:03:49.230 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.230 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.230 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.230 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.230 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092233+0000", 2024-04-19T02:03:49.230 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.230 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.231 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.231 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092233+0000", 2024-04-19T02:03:49.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8699999999999995e-07 2024-04-19T02:03:49.231 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.231 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.231 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092282+0000", 2024-04-19T02:03:49.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8869999999999998e-05 2024-04-19T02:03:49.231 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.231 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.231 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092311+0000", 2024-04-19T02:03:49.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9201e-05 2024-04-19T02:03:49.231 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.231 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.232 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092342+0000", 2024-04-19T02:03:49.232 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1275000000000001e-05 2024-04-19T02:03:49.232 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.232 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.232 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.232 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092597+0000", 2024-04-19T02:03:49.232 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00025494899999999997 2024-04-19T02:03:49.232 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.232 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.232 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:49.232 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092622+0000", 2024-04-19T02:03:49.232 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4355e-05 2024-04-19T02:03:49.232 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.232 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.232 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:49.232 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.118397+0000", 2024-04-19T02:03:49.232 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.025775005 2024-04-19T02:03:49.233 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.233 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.233 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.233 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.118442+0000", 2024-04-19T02:03:49.233 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5123000000000001e-05 2024-04-19T02:03:49.233 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.233 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.233 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.233 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.233 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.233 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:462 3.3s0 3:c26f510a:::smithi05930959-46:head [read 0~770048,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:03:49.233 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.138590+0000", 2024-04-19T02:03:49.233 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 18.047955924, 2024-04-19T02:03:49.233 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011711944, 2024-04-19T02:03:49.233 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.233 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.234 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:49.234 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:49.234 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:49.234 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:49.234 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 462 2024-04-19T02:03:49.234 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.234 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.234 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.234 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.234 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138590+0000", 2024-04-19T02:03:49.234 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.234 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.234 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.234 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.235 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138590+0000", 2024-04-19T02:03:49.235 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.235 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.235 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.235 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.235 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138591+0000", 2024-04-19T02:03:49.235 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1100000000000005e-07 2024-04-19T02:03:49.235 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.235 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.235 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.235 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138596+0000", 2024-04-19T02:03:49.235 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6740000000000002e-06 2024-04-19T02:03:49.235 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.235 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.235 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.235 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138611+0000", 2024-04-19T02:03:49.235 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4425e-05 2024-04-19T02:03:49.236 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.236 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.236 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138619+0000", 2024-04-19T02:03:49.236 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1880000000000002e-06 2024-04-19T02:03:49.236 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.236 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.236 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.139793+0000", 2024-04-19T02:03:49.236 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001174247 2024-04-19T02:03:49.236 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.236 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:49.236 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.140277+0000", 2024-04-19T02:03:49.236 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00048428099999999998 2024-04-19T02:03:49.236 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.237 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.237 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.237 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.150302+0000", 2024-04-19T02:03:49.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010024318000000001 2024-04-19T02:03:49.237 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.237 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.237 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.237 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.237 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.237 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:463 3.3s0 3:c26f510a:::smithi05930959-46:head [read 0~770048,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:03:49.237 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.138651+0000", 2024-04-19T02:03:49.237 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 18.047895048000001, 2024-04-19T02:03:49.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012901299999999999, 2024-04-19T02:03:49.237 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.237 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.237 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:49.237 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:49.238 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:49.238 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:49.238 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 463 2024-04-19T02:03:49.238 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.238 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.238 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.238 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.238 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138651+0000", 2024-04-19T02:03:49.238 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.238 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.238 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.238 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.238 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138651+0000", 2024-04-19T02:03:49.238 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.239 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.239 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.239 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138651+0000", 2024-04-19T02:03:49.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4000000000000001e-07 2024-04-19T02:03:49.239 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.239 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.239 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138657+0000", 2024-04-19T02:03:49.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9479999999999996e-06 2024-04-19T02:03:49.239 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.239 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.239 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138667+0000", 2024-04-19T02:03:49.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8679999999999994e-06 2024-04-19T02:03:49.239 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.240 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.240 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.138675+0000", 2024-04-19T02:03:49.240 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4989999999999997e-06 2024-04-19T02:03:49.240 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.240 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.240 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.240 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.141119+0000", 2024-04-19T02:03:49.240 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0024445019999999999 2024-04-19T02:03:49.240 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.240 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.240 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:49.240 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.141379+0000", 2024-04-19T02:03:49.240 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000259595 2024-04-19T02:03:49.240 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.240 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.240 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.240 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.151552+0000", 2024-04-19T02:03:49.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010173148 2024-04-19T02:03:49.241 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.241 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.241 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.241 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.241 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.241 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:582 3.3s0 3:c26f510a:::smithi05930959-46:head [append 0~671744] snapc 10=[10,f,c] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-19T02:03:49.241 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:40.208006+0000", 2024-04-19T02:03:49.241 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 8.9785395769999994, 2024-04-19T02:03:49.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022938303, 2024-04-19T02:03:49.241 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.241 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.241 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:49.241 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:49.241 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:49.241 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:49.241 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 582 2024-04-19T02:03:49.242 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.242 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.242 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.242 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.242 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.208006+0000", 2024-04-19T02:03:49.242 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.242 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.242 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.242 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.242 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.208006+0000", 2024-04-19T02:03:49.242 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.242 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.242 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.242 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.243 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.208009+0000", 2024-04-19T02:03:49.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2930000000000001e-06 2024-04-19T02:03:49.243 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.243 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.243 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.243 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.209365+0000", 2024-04-19T02:03:49.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001355745 2024-04-19T02:03:49.243 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.243 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.243 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.243 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.209399+0000", 2024-04-19T02:03:49.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3921000000000003e-05 2024-04-19T02:03:49.243 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.243 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.243 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.243 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.209414+0000", 2024-04-19T02:03:49.244 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4609e-05 2024-04-19T02:03:49.244 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.244 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.244 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.209587+0000", 2024-04-19T02:03:49.244 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017337400000000001 2024-04-19T02:03:49.244 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.244 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:49.244 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.209916+0000", 2024-04-19T02:03:49.244 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00032879699999999998 2024-04-19T02:03:49.244 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.244 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:49.244 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.212846+0000", 2024-04-19T02:03:49.245 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002929942 2024-04-19T02:03:49.245 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.245 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.245 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:49.245 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.230190+0000", 2024-04-19T02:03:49.245 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017344259000000001 2024-04-19T02:03:49.245 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.245 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.245 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:49.245 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.230733+0000", 2024-04-19T02:03:49.245 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00054272499999999998 2024-04-19T02:03:49.245 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.245 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.245 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.245 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.230944+0000", 2024-04-19T02:03:49.245 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000211638 2024-04-19T02:03:49.246 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.246 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.246 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.246 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.246 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.246 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:583 3.3s0 3:c26f510a:::smithi05930959-46:head [append 0~753664] snapc 10=[10,f,c] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-19T02:03:49.246 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:40.213899+0000", 2024-04-19T02:03:49.246 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 8.9726471980000007, 2024-04-19T02:03:49.246 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018638891000000001, 2024-04-19T02:03:49.246 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.246 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.246 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:49.246 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:49.246 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:49.246 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:49.247 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 583 2024-04-19T02:03:49.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.247 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.213899+0000", 2024-04-19T02:03:49.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.213899+0000", 2024-04-19T02:03:49.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.213902+0000", 2024-04-19T02:03:49.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.653e-06 2024-04-19T02:03:49.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.215557+0000", 2024-04-19T02:03:49.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0016552800000000001 2024-04-19T02:03:49.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.215599+0000", 2024-04-19T02:03:49.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1111e-05 2024-04-19T02:03:49.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.215623+0000", 2024-04-19T02:03:49.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4834999999999999e-05 2024-04-19T02:03:49.249 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.249 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.249 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.216512+0000", 2024-04-19T02:03:49.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00088845700000000005 2024-04-19T02:03:49.249 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.249 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:49.249 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.216771+0000", 2024-04-19T02:03:49.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00025874300000000003 2024-04-19T02:03:49.249 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.249 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:49.249 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.219870+0000", 2024-04-19T02:03:49.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0030994759999999999 2024-04-19T02:03:49.250 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:49.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.230345+0000", 2024-04-19T02:03:49.250 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010474778000000001 2024-04-19T02:03:49.250 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:49.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.232336+0000", 2024-04-19T02:03:49.250 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0019915110000000001 2024-04-19T02:03:49.250 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.232537+0000", 2024-04-19T02:03:49.250 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020104699999999999 2024-04-19T02:03:49.251 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.251 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.251 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.251 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.251 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.251 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:584 3.3s0 3:c26f510a:::smithi05930959-46:head [append 0~409600] snapc 10=[10,f,c] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-19T02:03:49.251 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:40.217468+0000", 2024-04-19T02:03:49.251 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 8.9690776060000008, 2024-04-19T02:03:49.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016044577000000001, 2024-04-19T02:03:49.251 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.251 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.251 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:49.251 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:49.251 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:49.251 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:49.252 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 584 2024-04-19T02:03:49.252 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.252 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.252 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.252 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.252 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.217468+0000", 2024-04-19T02:03:49.252 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.252 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.252 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.252 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.252 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.217468+0000", 2024-04-19T02:03:49.252 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.252 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.252 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.252 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.217471+0000", 2024-04-19T02:03:49.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7360000000000001e-06 2024-04-19T02:03:49.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218367+0000", 2024-04-19T02:03:49.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00089619299999999999 2024-04-19T02:03:49.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218402+0000", 2024-04-19T02:03:49.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4708e-05 2024-04-19T02:03:49.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.254 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218440+0000", 2024-04-19T02:03:49.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8362000000000003e-05 2024-04-19T02:03:49.254 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.254 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.254 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.254 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.222804+0000", 2024-04-19T02:03:49.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.004363596 2024-04-19T02:03:49.254 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.254 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.254 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:49.254 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.223100+0000", 2024-04-19T02:03:49.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00029668099999999997 2024-04-19T02:03:49.254 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.254 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.254 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:49.254 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.225367+0000", 2024-04-19T02:03:49.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0022668010000000001 2024-04-19T02:03:49.255 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:49.255 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.230348+0000", 2024-04-19T02:03:49.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0049803299999999998 2024-04-19T02:03:49.255 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:49.255 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.233363+0000", 2024-04-19T02:03:49.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0030157410000000002 2024-04-19T02:03:49.255 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.255 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.233513+0000", 2024-04-19T02:03:49.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000149429 2024-04-19T02:03:49.256 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.256 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.256 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.256 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:585 3.3s0 3:c26f510a:::smithi05930959-46:head [setxattr _header (57) in=7b] snapc 10=[10,f,c] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-19T02:03:49.256 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:40.218475+0000", 2024-04-19T02:03:49.256 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 8.9680702459999999, 2024-04-19T02:03:49.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017451903000000001, 2024-04-19T02:03:49.256 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.256 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.256 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:49.256 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:49.256 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:49.256 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:49.256 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 585 2024-04-19T02:03:49.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.257 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218475+0000", 2024-04-19T02:03:49.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218475+0000", 2024-04-19T02:03:49.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218477+0000", 2024-04-19T02:03:49.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.66e-06 2024-04-19T02:03:49.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218486+0000", 2024-04-19T02:03:49.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9530000000000005e-06 2024-04-19T02:03:49.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218508+0000", 2024-04-19T02:03:49.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2294999999999999e-05 2024-04-19T02:03:49.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218523+0000", 2024-04-19T02:03:49.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.488e-05 2024-04-19T02:03:49.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.259 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.259 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.227831+0000", 2024-04-19T02:03:49.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.009308205 2024-04-19T02:03:49.259 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.259 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:49.259 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.228114+0000", 2024-04-19T02:03:49.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028219900000000001 2024-04-19T02:03:49.259 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.259 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:49.259 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.229580+0000", 2024-04-19T02:03:49.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001466799 2024-04-19T02:03:49.259 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.260 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:49.260 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.235272+0000", 2024-04-19T02:03:49.260 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0056910570000000002 2024-04-19T02:03:49.260 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.260 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:49.260 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.235624+0000", 2024-04-19T02:03:49.260 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00035226799999999998 2024-04-19T02:03:49.260 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.260 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.260 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.235927+0000", 2024-04-19T02:03:49.260 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00030358700000000002 2024-04-19T02:03:49.260 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.260 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.260 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.261 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.261 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:586 3.3s0 3:c26f510a:::smithi05930959-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56)", 2024-04-19T02:03:49.261 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:40.218545+0000", 2024-04-19T02:03:49.261 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 8.9680011260000008, 2024-04-19T02:03:49.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.023206135999999999, 2024-04-19T02:03:49.261 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.261 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.261 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:49.261 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:49.261 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:49.261 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:49.261 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 586 2024-04-19T02:03:49.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.261 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218545+0000", 2024-04-19T02:03:49.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218545+0000", 2024-04-19T02:03:49.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218546+0000", 2024-04-19T02:03:49.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4160000000000001e-06 2024-04-19T02:03:49.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218553+0000", 2024-04-19T02:03:49.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1770000000000001e-06 2024-04-19T02:03:49.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218577+0000", 2024-04-19T02:03:49.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3635e-05 2024-04-19T02:03:49.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218593+0000", 2024-04-19T02:03:49.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5892999999999998e-05 2024-04-19T02:03:49.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.264 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.264 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.234991+0000", 2024-04-19T02:03:49.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016397954999999999 2024-04-19T02:03:49.264 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.264 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.264 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:49.264 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.235250+0000", 2024-04-19T02:03:49.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000259547 2024-04-19T02:03:49.264 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.264 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.264 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.264 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.236766+0000", 2024-04-19T02:03:49.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0015157599999999999 2024-04-19T02:03:49.264 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.264 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.264 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:49.265 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.237008+0000", 2024-04-19T02:03:49.265 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00024161300000000001 2024-04-19T02:03:49.265 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.265 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.265 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.265 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.241751+0000", 2024-04-19T02:03:49.265 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0047431399999999999 2024-04-19T02:03:49.265 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.265 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.265 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.265 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.265 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.265 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:703 3.5s0 3:aa7bfa0e:::smithi05930959-37:head [delete] snapc 14=[14,13,12,10,c] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e62)", 2024-04-19T02:03:49.265 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:46.467970+0000", 2024-04-19T02:03:49.265 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 2.718575585, 2024-04-19T02:03:49.266 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012271491000000001, 2024-04-19T02:03:49.266 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.266 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.266 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:49.266 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:49.266 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:49.266 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:49.266 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 703 2024-04-19T02:03:49.266 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.266 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.266 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.266 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.266 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.467970+0000", 2024-04-19T02:03:49.266 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.266 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.467970+0000", 2024-04-19T02:03:49.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.467974+0000", 2024-04-19T02:03:49.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7670000000000001e-06 2024-04-19T02:03:49.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.467993+0000", 2024-04-19T02:03:49.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9225999999999999e-05 2024-04-19T02:03:49.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.268 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.468037+0000", 2024-04-19T02:03:49.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4094000000000002e-05 2024-04-19T02:03:49.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.268 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.268 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.468063+0000", 2024-04-19T02:03:49.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5556000000000001e-05 2024-04-19T02:03:49.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.268 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.268 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.468240+0000", 2024-04-19T02:03:49.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000177682 2024-04-19T02:03:49.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.268 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:49.268 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.468452+0000", 2024-04-19T02:03:49.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021193699999999999 2024-04-19T02:03:49.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.269 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:49.269 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.473618+0000", 2024-04-19T02:03:49.269 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0051658320000000004 2024-04-19T02:03:49.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.269 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:49.269 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.476693+0000", 2024-04-19T02:03:49.269 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0030752459999999998 2024-04-19T02:03:49.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.269 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:49.269 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.479958+0000", 2024-04-19T02:03:49.269 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0032646340000000002 2024-04-19T02:03:49.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.270 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.270 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.480242+0000", 2024-04-19T02:03:49.270 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000283517 2024-04-19T02:03:49.270 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.270 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.270 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.270 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.270 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.270 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:49.271 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-19T02:03:49.272 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:49.274+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:49.373 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:49.369+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:49.392 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:49.392 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:03:49.392 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-04-19T02:03:49.392 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:49.406 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-19T02:03:49.474 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:49.473+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:49.570 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:49.571 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:03:49.571 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-04-19T02:03:49.571 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-04-19T02:03:49.571 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:49.575 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:49.573+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:49.584 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-19T02:03:49.676 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:49.678+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:49.751 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:49.751 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-04-19T02:03:49.752 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-04-19T02:03:49.752 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-04-19T02:03:49.752 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.752 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/26 ECSubWrite(tid=132, reqid=osd.3.0:132, at_version=42'63, trim_to=0'0, roll_forward_to=40'61))", 2024-04-19T02:03:49.752 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.748602+0000", 2024-04-19T02:03:49.752 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 23.003088773999998, 2024-04-19T02:03:49.752 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.024032777000000002, 2024-04-19T02:03:49.752 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.752 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.752 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:49.752 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.752 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.752 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.752 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748602+0000", 2024-04-19T02:03:49.752 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.753 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.753 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.753 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.753 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748602+0000", 2024-04-19T02:03:49.753 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.753 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.753 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.753 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.753 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748603+0000", 2024-04-19T02:03:49.753 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4600000000000003e-07 2024-04-19T02:03:49.753 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.753 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.753 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.753 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748657+0000", 2024-04-19T02:03:49.753 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3803e-05 2024-04-19T02:03:49.753 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.754 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.754 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.754 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748688+0000", 2024-04-19T02:03:49.754 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0871000000000003e-05 2024-04-19T02:03:49.754 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.754 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.754 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.754 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748710+0000", 2024-04-19T02:03:49.754 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2634999999999999e-05 2024-04-19T02:03:49.754 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.754 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.754 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.754 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748980+0000", 2024-04-19T02:03:49.754 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00026984900000000001 2024-04-19T02:03:49.754 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.755 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.755 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:49.755 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749007+0000", 2024-04-19T02:03:49.755 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6225e-05 2024-04-19T02:03:49.755 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.755 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.755 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:49.755 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.772533+0000", 2024-04-19T02:03:49.755 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.023526782 2024-04-19T02:03:49.755 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.755 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.755 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.755 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.772635+0000", 2024-04-19T02:03:49.755 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000101766 2024-04-19T02:03:49.755 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.755 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.755 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.756 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.756 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.756 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.8s3 42/26 ECSubWrite(tid=71, reqid=osd.4.0:71, at_version=42'33, trim_to=0'0, roll_forward_to=37'31))", 2024-04-19T02:03:49.756 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.751483+0000", 2024-04-19T02:03:49.756 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 23.000208321999999, 2024-04-19T02:03:49.756 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.024697106999999999, 2024-04-19T02:03:49.756 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.756 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.756 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:49.756 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.756 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.756 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.756 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751483+0000", 2024-04-19T02:03:49.756 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.756 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.756 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.757 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.757 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751483+0000", 2024-04-19T02:03:49.757 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.757 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.757 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.757 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.757 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751483+0000", 2024-04-19T02:03:49.757 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9999999999999997e-07 2024-04-19T02:03:49.757 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.757 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.757 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.757 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751545+0000", 2024-04-19T02:03:49.757 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1958999999999994e-05 2024-04-19T02:03:49.757 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.757 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.757 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.757 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755035+0000", 2024-04-19T02:03:49.758 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0034897819999999999 2024-04-19T02:03:49.758 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.758 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.758 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.758 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755064+0000", 2024-04-19T02:03:49.758 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9227999999999999e-05 2024-04-19T02:03:49.758 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.758 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.758 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.758 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755228+0000", 2024-04-19T02:03:49.758 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016367700000000001 2024-04-19T02:03:49.758 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.758 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.758 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:49.758 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755256+0000", 2024-04-19T02:03:49.758 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7900999999999999e-05 2024-04-19T02:03:49.759 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.759 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.759 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:49.759 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.776106+0000", 2024-04-19T02:03:49.759 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020850560000000001 2024-04-19T02:03:49.759 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.759 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.759 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.759 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.776180+0000", 2024-04-19T02:03:49.759 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3399999999999995e-05 2024-04-19T02:03:49.759 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.759 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.759 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.759 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.759 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.759 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/26 ECSubWrite(tid=134, reqid=osd.3.0:134, at_version=42'65, trim_to=0'0, roll_forward_to=40'61))", 2024-04-19T02:03:49.759 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.751576+0000", 2024-04-19T02:03:49.760 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 23.000114615000001, 2024-04-19T02:03:49.760 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.027269287, 2024-04-19T02:03:49.760 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.760 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.760 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:49.760 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.760 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.760 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.760 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751576+0000", 2024-04-19T02:03:49.760 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.760 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.760 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.760 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.760 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751576+0000", 2024-04-19T02:03:49.760 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.760 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.761 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.761 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.761 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751578+0000", 2024-04-19T02:03:49.761 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9659999999999999e-06 2024-04-19T02:03:49.761 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.761 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.761 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.761 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751651+0000", 2024-04-19T02:03:49.761 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2781000000000007e-05 2024-04-19T02:03:49.761 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.761 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.761 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.761 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755528+0000", 2024-04-19T02:03:49.761 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0038770359999999999 2024-04-19T02:03:49.761 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.761 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.761 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.762 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755558+0000", 2024-04-19T02:03:49.762 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9414e-05 2024-04-19T02:03:49.762 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.762 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.762 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.762 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755853+0000", 2024-04-19T02:03:49.762 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00029508699999999998 2024-04-19T02:03:49.762 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.762 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.762 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:49.762 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755881+0000", 2024-04-19T02:03:49.762 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7965000000000001e-05 2024-04-19T02:03:49.762 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.762 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.762 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:49.763 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778764+0000", 2024-04-19T02:03:49.763 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022883244000000001 2024-04-19T02:03:49.763 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.763 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.763 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.763 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778846+0000", 2024-04-19T02:03:49.763 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1793999999999996e-05 2024-04-19T02:03:49.763 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.763 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.763 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.763 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.763 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.763 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.es3 42/26 ECSubWrite(tid=32, reqid=osd.7.0:32, at_version=42'25, trim_to=0'0, roll_forward_to=40'23))", 2024-04-19T02:03:49.763 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.755373+0000", 2024-04-19T02:03:49.763 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 22.996318299999999, 2024-04-19T02:03:49.763 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020691993999999998, 2024-04-19T02:03:49.763 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.764 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.764 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:49.764 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.764 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.764 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.764 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755373+0000", 2024-04-19T02:03:49.764 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.764 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.764 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.764 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.764 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755373+0000", 2024-04-19T02:03:49.764 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.764 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.764 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.764 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.764 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755373+0000", 2024-04-19T02:03:49.765 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5300000000000004e-07 2024-04-19T02:03:49.765 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.765 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.765 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.765 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755450+0000", 2024-04-19T02:03:49.765 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6663999999999996e-05 2024-04-19T02:03:49.765 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.765 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.765 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.765 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755476+0000", 2024-04-19T02:03:49.765 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6395000000000001e-05 2024-04-19T02:03:49.765 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.765 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.765 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.765 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755496+0000", 2024-04-19T02:03:49.765 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0018e-05 2024-04-19T02:03:49.766 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.766 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.766 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.766 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.757829+0000", 2024-04-19T02:03:49.766 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002333052 2024-04-19T02:03:49.766 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.766 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.766 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:49.766 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.757857+0000", 2024-04-19T02:03:49.766 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7317999999999998e-05 2024-04-19T02:03:49.766 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.766 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.766 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:49.766 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.775968+0000", 2024-04-19T02:03:49.766 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018111454999999999 2024-04-19T02:03:49.766 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.767 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.767 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.767 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.776065+0000", 2024-04-19T02:03:49.767 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6539000000000005e-05 2024-04-19T02:03:49.767 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.767 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.767 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.767 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.767 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.767 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:371 3.as0 3:54128c7a:::smithi05930959-20:head [read 0~1605632,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:49.767 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.760138+0000", 2024-04-19T02:03:49.767 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 22.991553184000001, 2024-04-19T02:03:49.767 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.031633367000000003, 2024-04-19T02:03:49.767 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.767 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.767 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:49.768 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:49.768 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:49.768 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:49.768 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 371 2024-04-19T02:03:49.768 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.768 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.768 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.768 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.768 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760138+0000", 2024-04-19T02:03:49.768 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.768 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.768 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.768 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.768 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760138+0000", 2024-04-19T02:03:49.768 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.768 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.768 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.769 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.769 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760141+0000", 2024-04-19T02:03:49.769 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0790000000000001e-06 2024-04-19T02:03:49.769 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.769 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.769 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.769 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760155+0000", 2024-04-19T02:03:49.769 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4083000000000001e-05 2024-04-19T02:03:49.769 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.769 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.769 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.769 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760187+0000", 2024-04-19T02:03:49.769 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1964000000000001e-05 2024-04-19T02:03:49.769 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.769 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.769 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.770 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760208+0000", 2024-04-19T02:03:49.770 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1395000000000001e-05 2024-04-19T02:03:49.770 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.770 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.770 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.770 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760524+0000", 2024-04-19T02:03:49.770 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00031564199999999998 2024-04-19T02:03:49.770 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.770 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.770 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:49.770 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760793+0000", 2024-04-19T02:03:49.770 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000268819 2024-04-19T02:03:49.770 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.770 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.770 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.770 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774492+0000", 2024-04-19T02:03:49.770 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013699065999999999 2024-04-19T02:03:49.771 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.771 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.771 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:49.771 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774730+0000", 2024-04-19T02:03:49.771 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00023788499999999999 2024-04-19T02:03:49.771 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.771 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.771 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.771 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.791771+0000", 2024-04-19T02:03:49.771 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017041434000000001 2024-04-19T02:03:49.771 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.771 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.771 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.771 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.771 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.771 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:372 3.as0 3:54128c7a:::smithi05930959-20:head [read 0~1605632,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:49.771 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.760260+0000", 2024-04-19T02:03:49.772 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 22.991430817000001, 2024-04-19T02:03:49.772 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.034139714000000002, 2024-04-19T02:03:49.772 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.772 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.772 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:49.772 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:49.772 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:49.772 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:49.772 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 372 2024-04-19T02:03:49.772 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.772 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.772 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.772 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.772 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760260+0000", 2024-04-19T02:03:49.772 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.772 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.772 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.773 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.773 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760260+0000", 2024-04-19T02:03:49.773 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.773 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.773 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.773 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.773 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760262+0000", 2024-04-19T02:03:49.773 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6029999999999999e-06 2024-04-19T02:03:49.773 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.773 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.773 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.773 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760272+0000", 2024-04-19T02:03:49.773 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6560000000000004e-06 2024-04-19T02:03:49.773 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.773 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.773 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.773 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760298+0000", 2024-04-19T02:03:49.774 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6625e-05 2024-04-19T02:03:49.774 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.774 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.774 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.774 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760316+0000", 2024-04-19T02:03:49.774 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7459e-05 2024-04-19T02:03:49.774 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.774 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.774 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.774 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760885+0000", 2024-04-19T02:03:49.774 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00056941200000000004 2024-04-19T02:03:49.774 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.774 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.774 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:49.774 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761126+0000", 2024-04-19T02:03:49.774 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000241124 2024-04-19T02:03:49.775 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.775 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.775 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.775 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.775702+0000", 2024-04-19T02:03:49.775 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014575764 2024-04-19T02:03:49.775 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.775 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.775 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:49.775 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.777674+0000", 2024-04-19T02:03:49.775 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001972297 2024-04-19T02:03:49.775 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.775 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.775 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.775 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.794400+0000", 2024-04-19T02:03:49.775 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016725773999999999 2024-04-19T02:03:49.775 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.775 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.776 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.776 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.776 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.776 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:373 3.as0 3:54128c7a:::smithi05930959-20:head [read 0~1605632,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:49.776 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.760341+0000", 2024-04-19T02:03:49.776 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 22.991350362999999, 2024-04-19T02:03:49.776 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.043665984999999997, 2024-04-19T02:03:49.776 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.776 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.776 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:49.776 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:49.776 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:49.776 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:49.776 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 373 2024-04-19T02:03:49.776 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.776 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.776 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.777 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.777 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760341+0000", 2024-04-19T02:03:49.777 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.777 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.777 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.777 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.777 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760341+0000", 2024-04-19T02:03:49.777 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.777 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.777 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.777 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.777 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760342+0000", 2024-04-19T02:03:49.777 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.497e-06 2024-04-19T02:03:49.777 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.777 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.777 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.778 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760350+0000", 2024-04-19T02:03:49.778 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1119999999999996e-06 2024-04-19T02:03:49.778 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.778 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.778 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.778 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760370+0000", 2024-04-19T02:03:49.778 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9219e-05 2024-04-19T02:03:49.778 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.778 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.778 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.778 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760385+0000", 2024-04-19T02:03:49.778 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5905000000000001e-05 2024-04-19T02:03:49.778 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.778 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.778 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.778 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761214+0000", 2024-04-19T02:03:49.779 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000828939 2024-04-19T02:03:49.779 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.779 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.779 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:49.779 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761439+0000", 2024-04-19T02:03:49.779 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022493900000000001 2024-04-19T02:03:49.779 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.779 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.779 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.779 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.794533+0000", 2024-04-19T02:03:49.779 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.033094077999999999 2024-04-19T02:03:49.779 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.779 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.779 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:49.779 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.794696+0000", 2024-04-19T02:03:49.779 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000162442 2024-04-19T02:03:49.779 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.780 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.780 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.780 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.804007+0000", 2024-04-19T02:03:49.780 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0093108540000000004 2024-04-19T02:03:49.780 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.780 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.780 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.780 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.780 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.780 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/26 ECSubWrite(tid=140, reqid=osd.3.0:134, at_version=0'0, trim_to=0'0, roll_forward_to=42'65))", 2024-04-19T02:03:49.780 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.779396+0000", 2024-04-19T02:03:49.780 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 22.972294601000002, 2024-04-19T02:03:49.780 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020804339000000002, 2024-04-19T02:03:49.780 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.780 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.780 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:49.780 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.780 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.781 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.781 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.779396+0000", 2024-04-19T02:03:49.781 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.781 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.781 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.781 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.781 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.779396+0000", 2024-04-19T02:03:49.781 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.781 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.781 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.781 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.781 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.779397+0000", 2024-04-19T02:03:49.781 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.9999999999999995e-07 2024-04-19T02:03:49.781 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.782 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.782 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.782 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.779421+0000", 2024-04-19T02:03:49.782 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3226000000000001e-05 2024-04-19T02:03:49.782 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.782 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.782 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.782 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.779468+0000", 2024-04-19T02:03:49.782 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7497000000000003e-05 2024-04-19T02:03:49.782 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.782 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.782 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.782 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.779487+0000", 2024-04-19T02:03:49.782 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9194999999999998e-05 2024-04-19T02:03:49.782 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.783 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.783 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.783 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.786927+0000", 2024-04-19T02:03:49.783 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0074392779999999997 2024-04-19T02:03:49.783 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.783 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.783 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:49.783 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.786949+0000", 2024-04-19T02:03:49.783 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2572999999999999e-05 2024-04-19T02:03:49.783 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.783 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.783 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:49.783 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.800145+0000", 2024-04-19T02:03:49.783 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013195475999999999 2024-04-19T02:03:49.783 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.784 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.784 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.784 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.800201+0000", 2024-04-19T02:03:49.784 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6094000000000002e-05 2024-04-19T02:03:49.784 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.784 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.784 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.784 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.784 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.784 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:381 3.as0 3:512144ec:::smithi05930959-48:head [create,setxattr pnc]aaY (11788) in=7b,setxattr ZTVVStUpk__YaS\\blR\\XgesYT[Or[[aUett_nil] (1859) in=40b,setxattr ZtotWOVcPP_oiTUUTlqtYZsNneQnmhc\\bNdNpsgo (1402) in=40b,setxattr cR\\akUitgYfao[oisZ^ach\\i_kdge\\[c_mbmUf]^ (313) in=40b,setxattr pRe[UcWhXR^oagaSYScYPQa]r[hdldO\\S`V^[nh^ (874) in=40b,setxattr srVXOoW^mbi\\ccUpc`barbsc (11788) in=24b,setxattr pqjhUkafUjRqdpObgi_ZX^QV (11788) in=24b,setxattr q]Of\\bd_VgdoaP\\k^tmmpUqq^otrVVOdrtgU]nVk (12280) in=40b,setxattr ]Ple__\\nrXPQYecmQXf_o^bgN`mofssSPQrYslSO (1859) in=40b,setxattr ZiQcjT`UZQ]es__dmo^h]NdV^`qT_hniQtVY`WsT (770) in=40b,setxattr gpT^oWmeYZpbVdojjWnUWO[YfZptiQhdWdhs]Zo^ (841) in=40b,setxattr `Zgeqf\\Rb][aUdTVXYhj[ealjcUnRaiY (11788) in=32b,setxattr [dN]aooYVPYS^Q`_`bqSNVSS^`ZT\\csYZSrnjtUT (11296) in=40b,setxattr ehcNktRfRjqWWi[^XWcZokN]Wb[YnkeNl\\lkt_t\\ (1262) in=40b,setxattr OdsZcnit[de]qNV (11788) in=15b,setxattr ^]^lenZo\\TjoZOpONRno]jtaWOd`tqNdWcYccWNk (560) in=40b,setxattr ZUohXUqoPmYaaTceWo\\SRigth\\qP[Tg_^^asjknX (1085) in=40b,setxattr lR`dUarlc[n`ctj^TlY\\QpagN_doPfV`lZQ^kNtY (526) in=40b,setxattr [iXNTskqfmN\\qVhqrm`rdXpaN_VaeRbd^O\\Z_gcY (1261) in=40b,setxattr t_jR_akNV_aXVR^icaga]cP]baNne_f_UdsoeVUk (1894) in=40b] snapc 9=[8,6] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T02:03:49.784 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:27.087092+0000", 2024-04-19T02:03:49.784 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 22.664599369000001, 2024-04-19T02:03:49.784 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.025861967999999999, 2024-04-19T02:03:49.784 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.784 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.785 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:49.785 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:49.785 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:49.785 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:49.785 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 381 2024-04-19T02:03:49.785 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.785 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.785 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.785 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.785 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087092+0000", 2024-04-19T02:03:49.785 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.785 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.785 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.785 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.785 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087092+0000", 2024-04-19T02:03:49.785 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.785 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.786 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.786 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.786 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087095+0000", 2024-04-19T02:03:49.786 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2799999999999999e-06 2024-04-19T02:03:49.786 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.786 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.786 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.786 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087306+0000", 2024-04-19T02:03:49.786 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021136000000000001 2024-04-19T02:03:49.786 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.786 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.786 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.786 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087341+0000", 2024-04-19T02:03:49.786 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4654999999999999e-05 2024-04-19T02:03:49.786 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.787 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.787 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.787 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087358+0000", 2024-04-19T02:03:49.787 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7261e-05 2024-04-19T02:03:49.787 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.787 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.787 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.787 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087536+0000", 2024-04-19T02:03:49.787 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000177424 2024-04-19T02:03:49.787 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.787 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.787 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op must wait for map", 2024-04-19T02:03:49.788 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087551+0000", 2024-04-19T02:03:49.788 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5362e-05 2024-04-19T02:03:49.788 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.788 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.788 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.788 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.092762+0000", 2024-04-19T02:03:49.788 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0052113589999999996 2024-04-19T02:03:49.788 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.788 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.788 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:49.788 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.092952+0000", 2024-04-19T02:03:49.788 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00018992100000000001 2024-04-19T02:03:49.788 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.788 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.788 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:49.788 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.095255+0000", 2024-04-19T02:03:49.789 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0023027030000000001 2024-04-19T02:03:49.789 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.789 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.789 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:49.789 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.101758+0000", 2024-04-19T02:03:49.789 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0065034380000000003 2024-04-19T02:03:49.789 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.789 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.789 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:49.789 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.112669+0000", 2024-04-19T02:03:49.789 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010910737 2024-04-19T02:03:49.789 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.789 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.789 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.789 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.112954+0000", 2024-04-19T02:03:49.789 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028446800000000001 2024-04-19T02:03:49.790 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.790 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.790 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.790 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.790 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.790 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:434 3.as0 3:5a72bad2:::smithi05930959-41:head [read 0~3293184,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:49.790 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:30.194592+0000", 2024-04-19T02:03:49.790 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 19.557099409999999, 2024-04-19T02:03:49.790 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020994932000000001, 2024-04-19T02:03:49.790 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.790 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.790 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:49.790 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:49.790 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:49.790 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:49.790 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 434 2024-04-19T02:03:49.791 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.791 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.791 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.791 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.791 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194592+0000", 2024-04-19T02:03:49.791 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.791 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.791 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.791 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.791 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194592+0000", 2024-04-19T02:03:49.791 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.791 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.791 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.791 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.791 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194594+0000", 2024-04-19T02:03:49.791 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6730000000000001e-06 2024-04-19T02:03:49.792 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.792 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.792 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.792 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194608+0000", 2024-04-19T02:03:49.792 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3382e-05 2024-04-19T02:03:49.792 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.792 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.792 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.792 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194638+0000", 2024-04-19T02:03:49.792 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0691000000000001e-05 2024-04-19T02:03:49.792 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.792 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.792 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.792 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194659+0000", 2024-04-19T02:03:49.792 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0182000000000001e-05 2024-04-19T02:03:49.792 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.793 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.793 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.793 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194837+0000", 2024-04-19T02:03:49.793 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017831800000000001 2024-04-19T02:03:49.793 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.793 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.793 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:49.793 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194983+0000", 2024-04-19T02:03:49.793 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014610299999999999 2024-04-19T02:03:49.793 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.793 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.793 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.793 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.215587+0000", 2024-04-19T02:03:49.793 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020603583000000002 2024-04-19T02:03:49.793 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.793 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.794 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.794 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.794 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.794 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:435 3.as0 3:5a72bad2:::smithi05930959-41:head [read 0~3293184,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:49.794 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:30.194714+0000", 2024-04-19T02:03:49.794 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 19.556977529000001, 2024-04-19T02:03:49.794 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.025824666, 2024-04-19T02:03:49.794 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.794 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.794 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:49.794 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:49.794 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:49.794 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:49.794 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 435 2024-04-19T02:03:49.794 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.794 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.795 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.795 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.795 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194714+0000", 2024-04-19T02:03:49.795 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.795 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.795 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.795 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.795 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194714+0000", 2024-04-19T02:03:49.795 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.795 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.795 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.795 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.795 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194715+0000", 2024-04-19T02:03:49.795 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.497e-06 2024-04-19T02:03:49.795 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.795 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.795 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.796 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194727+0000", 2024-04-19T02:03:49.796 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2021999999999999e-05 2024-04-19T02:03:49.796 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.796 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.796 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.796 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194772+0000", 2024-04-19T02:03:49.796 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4620000000000003e-05 2024-04-19T02:03:49.796 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.796 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.796 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.796 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194790+0000", 2024-04-19T02:03:49.796 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7846000000000002e-05 2024-04-19T02:03:49.796 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.796 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.796 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.796 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.195633+0000", 2024-04-19T02:03:49.796 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00084348400000000003 2024-04-19T02:03:49.797 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.797 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.797 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:49.797 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.195885+0000", 2024-04-19T02:03:49.797 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00025215900000000001 2024-04-19T02:03:49.797 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.797 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.797 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.797 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.220538+0000", 2024-04-19T02:03:49.797 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.024653037999999999 2024-04-19T02:03:49.797 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.797 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.797 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.797 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.798 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.798 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:436 3.as0 3:5a72bad2:::smithi05930959-41:head [read 0~3293184,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:49.798 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:30.194819+0000", 2024-04-19T02:03:49.798 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 19.556872066, 2024-04-19T02:03:49.798 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.028970988999999999, 2024-04-19T02:03:49.798 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.798 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.798 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:49.798 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:49.798 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:49.798 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:49.798 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 436 2024-04-19T02:03:49.798 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.798 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.798 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.798 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.799 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194819+0000", 2024-04-19T02:03:49.799 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.799 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.799 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.799 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.799 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194819+0000", 2024-04-19T02:03:49.799 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.799 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.799 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.799 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.799 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194820+0000", 2024-04-19T02:03:49.799 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1829999999999999e-06 2024-04-19T02:03:49.799 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.799 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.799 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.799 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194828+0000", 2024-04-19T02:03:49.799 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.306e-06 2024-04-19T02:03:49.800 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.800 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.800 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.800 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194846+0000", 2024-04-19T02:03:49.800 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8592000000000001e-05 2024-04-19T02:03:49.800 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.800 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.800 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.800 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194860+0000", 2024-04-19T02:03:49.800 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4185999999999999e-05 2024-04-19T02:03:49.800 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.800 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.800 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.800 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198124+0000", 2024-04-19T02:03:49.800 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0032639430000000001 2024-04-19T02:03:49.800 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.800 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.801 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:49.801 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198217+0000", 2024-04-19T02:03:49.801 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2303e-05 2024-04-19T02:03:49.801 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.801 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.801 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.801 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.223790+0000", 2024-04-19T02:03:49.801 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.025573476000000001 2024-04-19T02:03:49.801 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.801 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.801 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.801 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.801 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.801 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s6 46/26 ECSubWrite(tid=163, reqid=client.4509.0:425, at_version=46'78, trim_to=0'0, roll_forward_to=46'75))", 2024-04-19T02:03:49.801 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:30.198808+0000", 2024-04-19T02:03:49.801 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 19.552883258000001, 2024-04-19T02:03:49.801 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021976591, 2024-04-19T02:03:49.802 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.802 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.802 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:49.802 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.802 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.802 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.802 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198808+0000", 2024-04-19T02:03:49.802 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.802 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.802 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.802 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.802 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198808+0000", 2024-04-19T02:03:49.802 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.802 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.802 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.802 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.803 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198809+0000", 2024-04-19T02:03:49.803 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6330000000000001e-06 2024-04-19T02:03:49.803 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.803 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.803 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.803 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198856+0000", 2024-04-19T02:03:49.803 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6414e-05 2024-04-19T02:03:49.803 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.803 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.803 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.803 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198882+0000", 2024-04-19T02:03:49.803 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6523999999999999e-05 2024-04-19T02:03:49.803 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.803 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.803 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.803 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198901+0000", 2024-04-19T02:03:49.803 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8763e-05 2024-04-19T02:03:49.804 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.804 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.804 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.804 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.202201+0000", 2024-04-19T02:03:49.804 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0033002190000000001 2024-04-19T02:03:49.804 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.804 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.804 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:49.804 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.202210+0000", 2024-04-19T02:03:49.804 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7830000000000004e-06 2024-04-19T02:03:49.804 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.804 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.804 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:49.804 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.220681+0000", 2024-04-19T02:03:49.804 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018470594999999999 2024-04-19T02:03:49.804 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.804 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.805 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.805 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.220784+0000", 2024-04-19T02:03:49.805 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010365999999999999 2024-04-19T02:03:49.805 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.805 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.805 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.805 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.805 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.805 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:500 3.fs0 3:fbe22738:::smithi05930959-7:b [read 0~2883584,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-04-19T02:03:49.805 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:32.048153+0000", 2024-04-19T02:03:49.805 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 17.703538427000002, 2024-04-19T02:03:49.805 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020543833000000001, 2024-04-19T02:03:49.805 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.805 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.805 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:49.805 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:49.806 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:49.806 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:49.806 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 500 2024-04-19T02:03:49.806 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.806 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.806 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.806 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.806 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048153+0000", 2024-04-19T02:03:49.806 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.806 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.806 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.806 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.806 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048153+0000", 2024-04-19T02:03:49.806 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.806 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.806 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.806 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.807 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048156+0000", 2024-04-19T02:03:49.807 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1269999999999999e-06 2024-04-19T02:03:49.807 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.807 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.807 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.807 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048173+0000", 2024-04-19T02:03:49.807 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6872e-05 2024-04-19T02:03:49.807 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.807 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.807 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.807 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048207+0000", 2024-04-19T02:03:49.807 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4246999999999999e-05 2024-04-19T02:03:49.807 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.807 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.807 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.807 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048233+0000", 2024-04-19T02:03:49.807 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5772e-05 2024-04-19T02:03:49.808 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.808 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.808 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.808 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048407+0000", 2024-04-19T02:03:49.808 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000174795 2024-04-19T02:03:49.808 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.808 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.808 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op must wait for map", 2024-04-19T02:03:49.808 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048430+0000", 2024-04-19T02:03:49.808 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2966e-05 2024-04-19T02:03:49.808 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.808 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.808 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.808 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.053551+0000", 2024-04-19T02:03:49.808 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0051201570000000002 2024-04-19T02:03:49.808 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.808 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.808 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:49.809 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.053843+0000", 2024-04-19T02:03:49.809 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00029192699999999999 2024-04-19T02:03:49.809 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.809 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.809 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.809 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.068696+0000", 2024-04-19T02:03:49.809 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014853969999999999 2024-04-19T02:03:49.809 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.809 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.809 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.809 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.809 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.809 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:501 3.fs0 3:fbe22738:::smithi05930959-7:b [read 0~2883584,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-04-19T02:03:49.809 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:32.048284+0000", 2024-04-19T02:03:49.809 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 17.703407489, 2024-04-19T02:03:49.809 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.02466438, 2024-04-19T02:03:49.809 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.809 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.810 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:49.810 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:49.810 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:49.810 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:49.810 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 501 2024-04-19T02:03:49.810 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.810 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.810 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.810 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.810 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048284+0000", 2024-04-19T02:03:49.810 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.810 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.810 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.810 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.810 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048284+0000", 2024-04-19T02:03:49.810 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.811 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.811 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.811 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.811 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048285+0000", 2024-04-19T02:03:49.811 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.513e-06 2024-04-19T02:03:49.811 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.811 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.811 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.811 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048294+0000", 2024-04-19T02:03:49.811 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4700000000000002e-06 2024-04-19T02:03:49.811 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.811 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.811 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.811 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048314+0000", 2024-04-19T02:03:49.811 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0424999999999998e-05 2024-04-19T02:03:49.811 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.811 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.812 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.812 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048329+0000", 2024-04-19T02:03:49.812 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5415999999999999e-05 2024-04-19T02:03:49.812 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.812 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.812 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.812 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048623+0000", 2024-04-19T02:03:49.812 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00029404700000000003 2024-04-19T02:03:49.812 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.812 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.812 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:03:49.812 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048637+0000", 2024-04-19T02:03:49.812 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3776e-05 2024-04-19T02:03:49.812 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.812 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.812 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.812 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.054889+0000", 2024-04-19T02:03:49.813 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.006251876 2024-04-19T02:03:49.813 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.813 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.813 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:49.813 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.055164+0000", 2024-04-19T02:03:49.813 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00027501199999999999 2024-04-19T02:03:49.813 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.813 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.813 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.813 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.072948+0000", 2024-04-19T02:03:49.813 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017783845 2024-04-19T02:03:49.813 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.813 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.813 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.813 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.813 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.813 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:502 3.fs0 3:fbe22738:::smithi05930959-7:b [read 0~2883584,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-04-19T02:03:49.814 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:32.048354+0000", 2024-04-19T02:03:49.814 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 17.703337352999998, 2024-04-19T02:03:49.814 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.035084603999999998, 2024-04-19T02:03:49.814 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.814 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.814 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:49.814 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:49.814 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:49.814 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:49.814 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 502 2024-04-19T02:03:49.814 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.814 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.814 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.814 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.814 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048354+0000", 2024-04-19T02:03:49.814 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.814 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.815 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.815 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.815 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048354+0000", 2024-04-19T02:03:49.815 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.815 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.815 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.815 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.815 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048355+0000", 2024-04-19T02:03:49.815 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1659999999999999e-06 2024-04-19T02:03:49.815 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.815 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.815 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.815 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048364+0000", 2024-04-19T02:03:49.815 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5829999999999996e-06 2024-04-19T02:03:49.815 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.815 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.815 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.815 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048383+0000", 2024-04-19T02:03:49.816 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8108999999999999e-05 2024-04-19T02:03:49.816 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.816 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.816 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.816 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048400+0000", 2024-04-19T02:03:49.816 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7051999999999999e-05 2024-04-19T02:03:49.816 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.816 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.816 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.816 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048713+0000", 2024-04-19T02:03:49.816 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00031348900000000002 2024-04-19T02:03:49.816 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.816 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.816 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:03:49.816 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048723+0000", 2024-04-19T02:03:49.816 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6760000000000002e-06 2024-04-19T02:03:49.816 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.817 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.817 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.817 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.056039+0000", 2024-04-19T02:03:49.817 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0073158099999999998 2024-04-19T02:03:49.817 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.817 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.817 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:49.817 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.056328+0000", 2024-04-19T02:03:49.817 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028893700000000002 2024-04-19T02:03:49.817 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.817 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.817 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.817 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.083438+0000", 2024-04-19T02:03:49.817 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.027110782 2024-04-19T02:03:49.817 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.817 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.817 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.818 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.818 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.818 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpReadReply(3.fs0 48/26 ECSubReadReply(tid=126, attrs_read=0))", 2024-04-19T02:03:49.818 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:32.063133+0000", 2024-04-19T02:03:49.818 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 17.688557918000001, 2024-04-19T02:03:49.818 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020419397999999998, 2024-04-19T02:03:49.818 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.818 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.818 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:49.818 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.818 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.818 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.818 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.063133+0000", 2024-04-19T02:03:49.818 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.818 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.819 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.819 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.819 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.063133+0000", 2024-04-19T02:03:49.819 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.819 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.819 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.819 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.819 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.063134+0000", 2024-04-19T02:03:49.819 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.136e-06 2024-04-19T02:03:49.819 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.819 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.819 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.819 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.069014+0000", 2024-04-19T02:03:49.819 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0058801340000000004 2024-04-19T02:03:49.819 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.819 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.820 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.820 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.069040+0000", 2024-04-19T02:03:49.820 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5599e-05 2024-04-19T02:03:49.820 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.820 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.820 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.820 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.069056+0000", 2024-04-19T02:03:49.820 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6095000000000001e-05 2024-04-19T02:03:49.820 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.820 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.820 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.820 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.076667+0000", 2024-04-19T02:03:49.820 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0076113300000000004 2024-04-19T02:03:49.820 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.820 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.820 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.820 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.083553+0000", 2024-04-19T02:03:49.821 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0068851040000000004 2024-04-19T02:03:49.821 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.821 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.821 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.821 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.821 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.821 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:535 3.fs0 3:f8a4544e:::smithi05930959-1:head [copy-from ver 31] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-19T02:03:49.821 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:36.137521+0000", 2024-04-19T02:03:49.821 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 13.614170323, 2024-04-19T02:03:49.821 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.030940776999999999, 2024-04-19T02:03:49.821 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.821 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.821 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:49.821 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:49.821 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:49.821 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:49.822 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 535 2024-04-19T02:03:49.822 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.822 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.822 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.822 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.822 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137521+0000", 2024-04-19T02:03:49.822 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.822 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.822 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.822 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.822 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137521+0000", 2024-04-19T02:03:49.822 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.822 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.822 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.822 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.823 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137524+0000", 2024-04-19T02:03:49.823 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3969999999999998e-06 2024-04-19T02:03:49.823 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.823 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.823 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.823 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137540+0000", 2024-04-19T02:03:49.823 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5619000000000001e-05 2024-04-19T02:03:49.823 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.823 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.823 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.823 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137571+0000", 2024-04-19T02:03:49.823 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0901000000000001e-05 2024-04-19T02:03:49.823 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.823 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.823 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.823 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137591+0000", 2024-04-19T02:03:49.824 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0279e-05 2024-04-19T02:03:49.824 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.824 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.824 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.824 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137849+0000", 2024-04-19T02:03:49.824 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000258535 2024-04-19T02:03:49.824 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.824 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.824 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op must wait for map", 2024-04-19T02:03:49.824 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137872+0000", 2024-04-19T02:03:49.824 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2507999999999999e-05 2024-04-19T02:03:49.824 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.824 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.824 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.824 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.141172+0000", 2024-04-19T02:03:49.824 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003300234 2024-04-19T02:03:49.824 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.824 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.825 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:49.825 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.141447+0000", 2024-04-19T02:03:49.825 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00027449700000000001 2024-04-19T02:03:49.825 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.825 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.825 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:49.825 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.161979+0000", 2024-04-19T02:03:49.825 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020531964 2024-04-19T02:03:49.825 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.825 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.825 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:49.825 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.165458+0000", 2024-04-19T02:03:49.825 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0034790060000000002 2024-04-19T02:03:49.825 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.825 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.825 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:49.826 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.168324+0000", 2024-04-19T02:03:49.826 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0028665280000000001 2024-04-19T02:03:49.826 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.826 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.826 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.826 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.168462+0000", 2024-04-19T02:03:49.826 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013730900000000001 2024-04-19T02:03:49.826 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.826 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.826 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.826 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.826 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.826 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:536 3.fs0 3:f8a4544e:::smithi05930959-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-04-19T02:03:49.826 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:36.137666+0000", 2024-04-19T02:03:49.826 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 13.614025135, 2024-04-19T02:03:49.827 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.027514961000000001, 2024-04-19T02:03:49.827 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.827 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.827 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:49.827 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:49.827 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:49.827 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:49.827 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 536 2024-04-19T02:03:49.827 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.827 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.827 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.827 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.827 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137666+0000", 2024-04-19T02:03:49.827 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.827 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.827 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.827 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.827 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137666+0000", 2024-04-19T02:03:49.828 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.828 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.828 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.828 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.828 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137668+0000", 2024-04-19T02:03:49.828 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9199999999999998e-06 2024-04-19T02:03:49.828 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.828 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.828 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.828 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137675+0000", 2024-04-19T02:03:49.828 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.153e-06 2024-04-19T02:03:49.828 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.828 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.828 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.828 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137697+0000", 2024-04-19T02:03:49.828 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1522000000000002e-05 2024-04-19T02:03:49.829 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.829 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.829 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.829 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137713+0000", 2024-04-19T02:03:49.829 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6016e-05 2024-04-19T02:03:49.829 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.829 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.829 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.829 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.138006+0000", 2024-04-19T02:03:49.829 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00029325899999999998 2024-04-19T02:03:49.829 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.829 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.829 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:03:49.829 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.138022+0000", 2024-04-19T02:03:49.829 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6563e-05 2024-04-19T02:03:49.829 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.829 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.830 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.830 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.141952+0000", 2024-04-19T02:03:49.830 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039293360000000003 2024-04-19T02:03:49.830 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.830 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.830 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for blocked object", 2024-04-19T02:03:49.830 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.142198+0000", 2024-04-19T02:03:49.830 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00024626500000000001 2024-04-19T02:03:49.830 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.830 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.830 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.830 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.164631+0000", 2024-04-19T02:03:49.830 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022432688999999999 2024-04-19T02:03:49.830 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.830 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.831 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:49.831 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.164884+0000", 2024-04-19T02:03:49.831 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000252892 2024-04-19T02:03:49.831 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.831 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.831 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.831 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.165181+0000", 2024-04-19T02:03:49.831 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00029734599999999998 2024-04-19T02:03:49.831 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.831 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.831 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.831 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.831 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.831 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:674 3.as0 3:54128c7a:::smithi05930959-20:head [read 0~3194880,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-19T02:03:49.831 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:46.325103+0000", 2024-04-19T02:03:49.831 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 3.426587676, 2024-04-19T02:03:49.831 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021675204999999999, 2024-04-19T02:03:49.832 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:49.832 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:49.832 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:49.832 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:49.832 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:49.832 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:49.832 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 674 2024-04-19T02:03:49.832 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.832 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:49.832 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.832 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:49.832 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.325103+0000", 2024-04-19T02:03:49.832 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.832 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.832 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.832 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:49.833 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.325103+0000", 2024-04-19T02:03:49.833 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:49.833 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.833 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.833 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:49.833 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.325105+0000", 2024-04-19T02:03:49.833 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4670000000000001e-06 2024-04-19T02:03:49.833 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.833 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.833 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:49.833 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.325114+0000", 2024-04-19T02:03:49.833 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0759999999999992e-06 2024-04-19T02:03:49.833 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.833 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.833 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:49.833 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.325134+0000", 2024-04-19T02:03:49.834 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9619e-05 2024-04-19T02:03:49.834 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.834 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.834 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:49.834 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.325148+0000", 2024-04-19T02:03:49.834 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4885999999999999e-05 2024-04-19T02:03:49.834 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.834 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.834 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:49.834 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.327186+0000", 2024-04-19T02:03:49.834 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0020380120000000001 2024-04-19T02:03:49.834 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.834 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.834 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:49.835 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.327378+0000", 2024-04-19T02:03:49.835 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00019130899999999999 2024-04-19T02:03:49.835 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:49.835 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:49.835 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:49.835 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.346779+0000", 2024-04-19T02:03:49.835 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019400836000000001 2024-04-19T02:03:49.835 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.835 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.835 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.835 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:49.835 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:49.835 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:49.837 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-19T02:03:49.837 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:49.837+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:49.938 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:49.937+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:49.973 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:49.973 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:03:49.973 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-04-19T02:03:49.973 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:49.988 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-19T02:03:50.039 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:50.041+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:50.055 INFO:tasks.rados.rados.0.smithi059.stdout:277: done (0 left) 2024-04-19T02:03:50.055 INFO:tasks.rados.rados.0.smithi059.stdout:278: rollback oid 40 current snap is 21 2024-04-19T02:03:50.055 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 40 to 14 2024-04-19T02:03:50.056 INFO:tasks.rados.rados.0.smithi059.stdout:279: read oid 3 snap 20 2024-04-19T02:03:50.056 INFO:tasks.rados.rados.0.smithi059.stdout:279: expect (ObjNum 108 snap 15 seq_num 108) 2024-04-19T02:03:50.056 INFO:tasks.rados.rados.0.smithi059.stdout:280: delete oid 44 current snap is 21 2024-04-19T02:03:50.071 INFO:tasks.rados.rados.0.smithi059.stdout:280: done (2 left) 2024-04-19T02:03:50.071 INFO:tasks.rados.rados.0.smithi059.stdout:281: read oid 45 snap 16 2024-04-19T02:03:50.071 INFO:tasks.rados.rados.0.smithi059.stdout:281: expect (ObjNum 93 snap 9 seq_num 93) 2024-04-19T02:03:50.071 INFO:tasks.rados.rados.0.smithi059.stdout:282: read oid 1 snap 14 2024-04-19T02:03:50.071 INFO:tasks.rados.rados.0.smithi059.stdout:282: expect (ObjNum 71 snap 5 seq_num 71) 2024-04-19T02:03:50.071 INFO:tasks.rados.rados.0.smithi059.stdout:283: rollback oid 26 current snap is 21 2024-04-19T02:03:50.071 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 26 to 10 2024-04-19T02:03:50.071 INFO:tasks.rados.rados.0.smithi059.stdout:284: read oid 35 snap -1 2024-04-19T02:03:50.071 INFO:tasks.rados.rados.0.smithi059.stdout:284: expect (ObjNum 34 snap 0 seq_num 34) 2024-04-19T02:03:50.072 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 16 current snap is 21 2024-04-19T02:03:50.072 INFO:tasks.rados.rados.0.smithi059.stdout:285: seq_num 134 ranges {1687552=704512,2392064=475136,2867200=573440,3440640=507904} 2024-04-19T02:03:50.090 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-19T02:03:50.098 INFO:tasks.rados.rados.0.smithi059.stdout:285: writing smithi05930959-16 from 1687552 to 2392064 tid 1 2024-04-19T02:03:50.103 INFO:tasks.rados.rados.0.smithi059.stdout:285: writing smithi05930959-16 from 2392064 to 2867200 tid 2 2024-04-19T02:03:50.110 INFO:tasks.rados.rados.0.smithi059.stdout:285: writing smithi05930959-16 from 2867200 to 3440640 tid 3 2024-04-19T02:03:50.116 INFO:tasks.rados.rados.0.smithi059.stdout:285: writing smithi05930959-16 from 3440640 to 3948544 tid 4 2024-04-19T02:03:50.116 INFO:tasks.rados.rados.0.smithi059.stdout:286: delete oid 17 current snap is 21 2024-04-19T02:03:50.116 INFO:tasks.rados.rados.0.smithi059.stdout:278: finishing rollback tid 0 to smithi05930959-40 2024-04-19T02:03:50.116 INFO:tasks.rados.rados.0.smithi059.stdout:278: finishing rollback tid 1 to smithi05930959-40 2024-04-19T02:03:50.116 INFO:tasks.rados.rados.0.smithi059.stdout:283: finishing rollback tid 0 to smithi05930959-26 2024-04-19T02:03:50.116 INFO:tasks.rados.rados.0.smithi059.stdout:283: finishing rollback tid 1 to smithi05930959-26 2024-04-19T02:03:50.116 INFO:tasks.rados.rados.0.smithi059.stdout:283: finishing rollback tid 2 to smithi05930959-26 2024-04-19T02:03:50.116 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 26 v 72 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-19T02:03:50.117 INFO:tasks.rados.rados.0.smithi059.stdout:284: expect (ObjNum 34 snap 0 seq_num 34) 2024-04-19T02:03:50.130 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:50.130 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:03:50.130 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-04-19T02:03:50.130 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-04-19T02:03:50.131 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:50.139 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:50.137+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:50.140 INFO:tasks.rados.rados.0.smithi059.stdout:282: expect (ObjNum 71 snap 5 seq_num 71) 2024-04-19T02:03:50.146 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-19T02:03:50.190 INFO:tasks.rados.rados.0.smithi059.stdout:278: finishing rollback tid 2 to smithi05930959-40 2024-04-19T02:03:50.190 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 40 v 67 (ObjNum 71 snap 5 seq_num 71) dirty exists 2024-04-19T02:03:50.190 INFO:tasks.rados.rados.0.smithi059.stdout:281: expect (ObjNum 93 snap 9 seq_num 93) 2024-04-19T02:03:50.232 INFO:tasks.rados.rados.0.smithi059.stdout:279: expect (ObjNum 108 snap 15 seq_num 108) 2024-04-19T02:03:50.240 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:50.237+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:50.312 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:50.312 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-04-19T02:03:50.312 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-04-19T02:03:50.312 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-04-19T02:03:50.312 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.313 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es4 35/26 ECSubWrite(tid=23, reqid=osd.7.0:23, at_version=35'18, trim_to=0'0, roll_forward_to=34'16))", 2024-04-19T02:03:50.313 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.676071+0000", 2024-04-19T02:03:50.313 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 30.636908059, 2024-04-19T02:03:50.313 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0084307070000000008, 2024-04-19T02:03:50.313 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.313 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.313 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:50.313 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.313 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.313 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.313 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.676071+0000", 2024-04-19T02:03:50.313 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.313 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.314 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.314 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.314 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.676071+0000", 2024-04-19T02:03:50.314 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.314 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.314 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.314 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.314 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.676072+0000", 2024-04-19T02:03:50.314 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7000000000000001e-07 2024-04-19T02:03:50.314 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.314 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.314 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.314 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.676098+0000", 2024-04-19T02:03:50.315 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6169999999999998e-05 2024-04-19T02:03:50.315 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.315 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.315 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.315 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.676110+0000", 2024-04-19T02:03:50.315 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2568000000000001e-05 2024-04-19T02:03:50.315 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.315 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.315 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.315 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.676122+0000", 2024-04-19T02:03:50.315 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1371e-05 2024-04-19T02:03:50.315 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.315 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.315 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.315 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.676504+0000", 2024-04-19T02:03:50.316 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00038213599999999999 2024-04-19T02:03:50.316 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.316 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.316 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:50.316 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.676513+0000", 2024-04-19T02:03:50.316 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.9509999999999995e-06 2024-04-19T02:03:50.316 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.316 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.316 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:50.316 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.684458+0000", 2024-04-19T02:03:50.316 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0079447419999999994 2024-04-19T02:03:50.316 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.316 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.316 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.316 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.684502+0000", 2024-04-19T02:03:50.317 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4499000000000002e-05 2024-04-19T02:03:50.317 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.317 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.317 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.317 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.317 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.317 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s7 35/26 ECSubWrite(tid=30, reqid=osd.1.0:29, at_version=0'0, trim_to=0'0, roll_forward_to=35'18))", 2024-04-19T02:03:50.317 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:19.680702+0000", 2024-04-19T02:03:50.317 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 30.632277405, 2024-04-19T02:03:50.317 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0072111470000000002, 2024-04-19T02:03:50.317 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.317 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.317 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:50.318 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.318 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.318 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.318 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680702+0000", 2024-04-19T02:03:50.318 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.318 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.318 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.318 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.318 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680702+0000", 2024-04-19T02:03:50.318 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.318 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.318 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.318 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.318 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680703+0000", 2024-04-19T02:03:50.318 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3099999999999996e-07 2024-04-19T02:03:50.319 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.319 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.319 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.319 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680717+0000", 2024-04-19T02:03:50.319 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4515000000000001e-05 2024-04-19T02:03:50.319 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.319 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.319 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.319 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680737+0000", 2024-04-19T02:03:50.319 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.984e-05 2024-04-19T02:03:50.319 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.319 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.319 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.319 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680747+0000", 2024-04-19T02:03:50.319 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.7410000000000004e-06 2024-04-19T02:03:50.319 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.320 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.320 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.320 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680846+0000", 2024-04-19T02:03:50.320 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.8572000000000004e-05 2024-04-19T02:03:50.320 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.320 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.320 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:50.320 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.680857+0000", 2024-04-19T02:03:50.320 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1055e-05 2024-04-19T02:03:50.320 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.320 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.320 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:50.320 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.687866+0000", 2024-04-19T02:03:50.320 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0070097149999999997 2024-04-19T02:03:50.320 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.321 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.321 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.321 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:19.687913+0000", 2024-04-19T02:03:50.321 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.6878000000000001e-05 2024-04-19T02:03:50.321 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.321 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.321 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.321 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.321 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.321 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(osd.9.18:2 3.1s0 3:874b3480:::smithi05930959-36:head [assert-version v6,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:50.321 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.775686+0000", 2024-04-19T02:03:50.321 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 25.537293590000001, 2024-04-19T02:03:50.321 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010480552000000001, 2024-04-19T02:03:50.321 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.321 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.322 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:50.322 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.322 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.322 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.322 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.775686+0000", 2024-04-19T02:03:50.322 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.322 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.322 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.322 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.322 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.775686+0000", 2024-04-19T02:03:50.322 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.322 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.322 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.322 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.323 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.775687+0000", 2024-04-19T02:03:50.323 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3659999999999999e-06 2024-04-19T02:03:50.323 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.323 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.323 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.323 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.775694+0000", 2024-04-19T02:03:50.323 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.2540000000000003e-06 2024-04-19T02:03:50.323 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.323 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.323 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.323 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.775706+0000", 2024-04-19T02:03:50.323 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1654000000000001e-05 2024-04-19T02:03:50.323 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.323 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.323 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.324 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.775715+0000", 2024-04-19T02:03:50.324 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0189999999999995e-06 2024-04-19T02:03:50.324 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.324 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.324 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.324 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.775831+0000", 2024-04-19T02:03:50.324 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000115606 2024-04-19T02:03:50.324 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.324 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.324 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:50.324 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.776060+0000", 2024-04-19T02:03:50.324 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00022901700000000001 2024-04-19T02:03:50.324 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.324 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.325 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.325 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.786166+0000", 2024-04-19T02:03:50.325 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010106636 2024-04-19T02:03:50.325 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.325 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.325 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.325 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.325 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.325 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.ds7 46/26 ECSubWrite(tid=187, reqid=osd.8.0:187, at_version=46'35, trim_to=0'0, roll_forward_to=45'31))", 2024-04-19T02:03:50.325 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.058419+0000", 2024-04-19T02:03:50.325 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 20.254560753, 2024-04-19T02:03:50.325 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.007118699, 2024-04-19T02:03:50.325 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.325 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.325 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:50.326 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.326 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.326 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.326 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.058419+0000", 2024-04-19T02:03:50.326 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.326 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.326 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.326 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.326 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.058419+0000", 2024-04-19T02:03:50.326 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.326 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.326 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.326 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.327 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.058420+0000", 2024-04-19T02:03:50.327 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.9999999999999996e-07 2024-04-19T02:03:50.327 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.327 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.327 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.327 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.058454+0000", 2024-04-19T02:03:50.327 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4678000000000002e-05 2024-04-19T02:03:50.327 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.327 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.327 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.327 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.058474+0000", 2024-04-19T02:03:50.327 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9439000000000001e-05 2024-04-19T02:03:50.327 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.327 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.327 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.328 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.058485+0000", 2024-04-19T02:03:50.328 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1646e-05 2024-04-19T02:03:50.328 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.328 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.328 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.328 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.058684+0000", 2024-04-19T02:03:50.328 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00019885500000000001 2024-04-19T02:03:50.328 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.328 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.328 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:50.328 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.058697+0000", 2024-04-19T02:03:50.328 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2571999999999999e-05 2024-04-19T02:03:50.328 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.328 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.328 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:50.329 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.065499+0000", 2024-04-19T02:03:50.329 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0068021770000000004 2024-04-19T02:03:50.329 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.329 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.329 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.329 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.065537+0000", 2024-04-19T02:03:50.329 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.8532000000000003e-05 2024-04-19T02:03:50.329 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.329 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.329 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.329 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.329 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.329 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s7 46/26 ECSubWrite(tid=58, reqid=osd.1.0:58, at_version=46'44, trim_to=0'0, roll_forward_to=45'42))", 2024-04-19T02:03:50.329 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.059818+0000", 2024-04-19T02:03:50.329 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 20.253161755000001, 2024-04-19T02:03:50.330 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012903721, 2024-04-19T02:03:50.330 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.330 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.330 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:50.330 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.330 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.330 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.330 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.059818+0000", 2024-04-19T02:03:50.330 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.330 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.330 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.330 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.330 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.059818+0000", 2024-04-19T02:03:50.330 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.331 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.331 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.331 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.331 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.059818+0000", 2024-04-19T02:03:50.331 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4900000000000002e-07 2024-04-19T02:03:50.331 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.331 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.331 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.331 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.059841+0000", 2024-04-19T02:03:50.331 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3317999999999999e-05 2024-04-19T02:03:50.331 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.331 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.331 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.331 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.066536+0000", 2024-04-19T02:03:50.332 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0066946920000000004 2024-04-19T02:03:50.332 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.332 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.332 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.332 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.066552+0000", 2024-04-19T02:03:50.332 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5758999999999999e-05 2024-04-19T02:03:50.332 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.332 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.332 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.332 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.066653+0000", 2024-04-19T02:03:50.332 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000101648 2024-04-19T02:03:50.332 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.332 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.332 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:50.332 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.066667+0000", 2024-04-19T02:03:50.333 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3648999999999999e-05 2024-04-19T02:03:50.333 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.333 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.333 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:50.333 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.072672+0000", 2024-04-19T02:03:50.333 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0060046709999999996 2024-04-19T02:03:50.333 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.333 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.333 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.333 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.072721+0000", 2024-04-19T02:03:50.333 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9734999999999997e-05 2024-04-19T02:03:50.333 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.333 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.333 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.333 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.334 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.334 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:415 3.1s0 3:874b3480:::smithi05930959-36:head [append 0~720896] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:50.334 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.072647+0000", 2024-04-19T02:03:50.334 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 20.240332695999999, 2024-04-19T02:03:50.334 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.020543788, 2024-04-19T02:03:50.334 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.334 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.334 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:50.334 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:50.334 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:50.334 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:50.334 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 415 2024-04-19T02:03:50.334 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.334 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.335 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.335 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.335 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.072647+0000", 2024-04-19T02:03:50.335 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.335 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.335 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.335 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.335 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.072647+0000", 2024-04-19T02:03:50.335 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.335 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.335 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.335 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.335 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.072649+0000", 2024-04-19T02:03:50.335 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3649999999999998e-06 2024-04-19T02:03:50.336 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.336 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.336 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.336 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.073285+0000", 2024-04-19T02:03:50.336 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00063577000000000002 2024-04-19T02:03:50.336 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.336 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.336 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.336 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.073308+0000", 2024-04-19T02:03:50.336 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2612e-05 2024-04-19T02:03:50.336 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.336 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.336 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.336 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.073319+0000", 2024-04-19T02:03:50.337 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1992e-05 2024-04-19T02:03:50.337 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.337 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.337 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.337 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.073482+0000", 2024-04-19T02:03:50.337 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00016297800000000001 2024-04-19T02:03:50.337 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.337 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.337 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:50.337 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.073685+0000", 2024-04-19T02:03:50.337 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00020296299999999999 2024-04-19T02:03:50.337 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.337 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.337 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:50.337 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.076845+0000", 2024-04-19T02:03:50.338 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0031591380000000001 2024-04-19T02:03:50.338 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.338 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.338 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:50.338 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.085769+0000", 2024-04-19T02:03:50.338 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0089239320000000007 2024-04-19T02:03:50.338 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.338 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.338 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:50.338 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.093097+0000", 2024-04-19T02:03:50.338 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0073282130000000001 2024-04-19T02:03:50.338 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.338 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.338 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.338 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.093191+0000", 2024-04-19T02:03:50.339 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3825000000000004e-05 2024-04-19T02:03:50.339 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.339 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.339 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.339 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.339 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.339 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:416 3.1s0 3:874b3480:::smithi05930959-36:head [append 0~442368] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:50.339 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.077896+0000", 2024-04-19T02:03:50.339 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 20.235083537000001, 2024-04-19T02:03:50.339 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.018242536, 2024-04-19T02:03:50.339 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.339 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.339 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:50.340 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:50.340 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:50.340 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:50.340 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 416 2024-04-19T02:03:50.340 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.340 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.340 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.340 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.340 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077896+0000", 2024-04-19T02:03:50.340 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.340 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.340 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.341 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.341 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077896+0000", 2024-04-19T02:03:50.341 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.341 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.341 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.341 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.341 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077898+0000", 2024-04-19T02:03:50.341 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1959999999999998e-06 2024-04-19T02:03:50.341 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.341 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.341 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.341 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.078421+0000", 2024-04-19T02:03:50.341 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00052314500000000003 2024-04-19T02:03:50.341 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.341 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.342 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.342 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.078441+0000", 2024-04-19T02:03:50.342 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9763999999999999e-05 2024-04-19T02:03:50.342 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.342 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.342 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.342 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.078453+0000", 2024-04-19T02:03:50.342 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1724e-05 2024-04-19T02:03:50.342 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.342 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.342 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.342 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.079580+0000", 2024-04-19T02:03:50.342 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0011274550000000001 2024-04-19T02:03:50.342 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.343 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.343 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:50.343 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.079689+0000", 2024-04-19T02:03:50.343 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108942 2024-04-19T02:03:50.343 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.343 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.343 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:50.343 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080934+0000", 2024-04-19T02:03:50.343 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001245187 2024-04-19T02:03:50.343 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.343 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.343 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:50.343 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.085771+0000", 2024-04-19T02:03:50.343 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.004836707 2024-04-19T02:03:50.343 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.344 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.344 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:50.344 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.096060+0000", 2024-04-19T02:03:50.344 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010288976 2024-04-19T02:03:50.344 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.344 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.344 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.344 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.096138+0000", 2024-04-19T02:03:50.344 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.8440000000000001e-05 2024-04-19T02:03:50.344 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.344 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.344 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.344 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.344 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.345 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:417 3.1s0 3:874b3480:::smithi05930959-36:head [append 0~327680] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:50.345 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.079192+0000", 2024-04-19T02:03:50.345 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 20.233787007, 2024-04-19T02:03:50.345 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.01712928, 2024-04-19T02:03:50.345 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.345 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.345 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:50.345 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:50.345 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:50.345 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:50.345 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 417 2024-04-19T02:03:50.345 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.345 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.345 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.346 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.346 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.079192+0000", 2024-04-19T02:03:50.346 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.346 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.346 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.346 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.346 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.079192+0000", 2024-04-19T02:03:50.346 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.346 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.346 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.346 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.346 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.079195+0000", 2024-04-19T02:03:50.346 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1880000000000001e-06 2024-04-19T02:03:50.346 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.347 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.347 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.347 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.079506+0000", 2024-04-19T02:03:50.347 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00031092000000000001 2024-04-19T02:03:50.347 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.347 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.347 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.347 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.079526+0000", 2024-04-19T02:03:50.347 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0404e-05 2024-04-19T02:03:50.347 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.347 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.347 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.347 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.079540+0000", 2024-04-19T02:03:50.347 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4233000000000001e-05 2024-04-19T02:03:50.347 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.348 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.348 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.348 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.081867+0000", 2024-04-19T02:03:50.348 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0023265030000000002 2024-04-19T02:03:50.348 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.348 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.348 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:50.348 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.082009+0000", 2024-04-19T02:03:50.348 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014187299999999999 2024-04-19T02:03:50.348 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.348 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.348 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:50.348 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.083078+0000", 2024-04-19T02:03:50.348 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0010689499999999999 2024-04-19T02:03:50.348 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.348 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.349 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:50.349 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.085773+0000", 2024-04-19T02:03:50.349 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002695771 2024-04-19T02:03:50.349 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.349 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.349 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:50.349 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.096256+0000", 2024-04-19T02:03:50.349 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010482551 2024-04-19T02:03:50.349 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.349 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.349 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.349 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.096322+0000", 2024-04-19T02:03:50.349 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.5887000000000001e-05 2024-04-19T02:03:50.349 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.349 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.350 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.350 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.350 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.350 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:418 3.1s0 3:874b3480:::smithi05930959-36:head [setxattr _header (56) in=7b] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:50.350 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.080040+0000", 2024-04-19T02:03:50.350 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 20.232939006999999, 2024-04-19T02:03:50.350 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.016869248, 2024-04-19T02:03:50.350 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.350 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.350 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:50.350 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:50.350 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:50.350 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:50.350 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 418 2024-04-19T02:03:50.351 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.351 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.351 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.351 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.351 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080040+0000", 2024-04-19T02:03:50.351 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.351 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.351 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.351 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.351 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080040+0000", 2024-04-19T02:03:50.351 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.351 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.351 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.351 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.351 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080042+0000", 2024-04-19T02:03:50.351 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.215e-06 2024-04-19T02:03:50.352 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.352 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.352 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.352 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080049+0000", 2024-04-19T02:03:50.352 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.3180000000000001e-06 2024-04-19T02:03:50.352 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.352 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.352 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.352 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080067+0000", 2024-04-19T02:03:50.352 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7805999999999999e-05 2024-04-19T02:03:50.352 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.352 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.352 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.352 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080078+0000", 2024-04-19T02:03:50.352 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1262e-05 2024-04-19T02:03:50.353 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.353 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.353 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.353 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.084057+0000", 2024-04-19T02:03:50.353 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0039794039999999998 2024-04-19T02:03:50.353 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.353 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.353 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:50.353 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.084161+0000", 2024-04-19T02:03:50.353 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000103862 2024-04-19T02:03:50.353 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.353 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.353 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:50.353 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.084626+0000", 2024-04-19T02:03:50.354 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00046457200000000001 2024-04-19T02:03:50.354 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.354 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.354 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:50.354 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.086690+0000", 2024-04-19T02:03:50.354 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0020638119999999999 2024-04-19T02:03:50.354 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.354 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.354 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:50.354 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.096836+0000", 2024-04-19T02:03:50.354 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.01014584 2024-04-19T02:03:50.354 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.354 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.354 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.355 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.096910+0000", 2024-04-19T02:03:50.355 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.4157000000000002e-05 2024-04-19T02:03:50.355 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.355 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.355 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.355 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.355 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.355 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:419 3.1s0 3:874b3480:::smithi05930959-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:50.355 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.080096+0000", 2024-04-19T02:03:50.355 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 20.232883782999998, 2024-04-19T02:03:50.355 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.019302687999999998, 2024-04-19T02:03:50.355 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.355 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.355 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:50.356 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:50.356 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:50.356 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:50.356 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 419 2024-04-19T02:03:50.356 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.356 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.356 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.356 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.356 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080096+0000", 2024-04-19T02:03:50.356 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.356 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.356 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.357 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.357 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080096+0000", 2024-04-19T02:03:50.357 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.357 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.357 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.357 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.357 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080096+0000", 2024-04-19T02:03:50.357 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.1200000000000002e-07 2024-04-19T02:03:50.357 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.357 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.357 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.357 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080100+0000", 2024-04-19T02:03:50.357 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.506e-06 2024-04-19T02:03:50.358 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.358 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.358 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.358 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080111+0000", 2024-04-19T02:03:50.358 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0879e-05 2024-04-19T02:03:50.358 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.358 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.358 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.358 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080119+0000", 2024-04-19T02:03:50.358 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.9629999999999998e-06 2024-04-19T02:03:50.358 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.358 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.358 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.359 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.085126+0000", 2024-04-19T02:03:50.359 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0050074280000000004 2024-04-19T02:03:50.359 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.359 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.359 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:50.359 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.085230+0000", 2024-04-19T02:03:50.359 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000103589 2024-04-19T02:03:50.359 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.359 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.359 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.359 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.097332+0000", 2024-04-19T02:03:50.359 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012101796999999999 2024-04-19T02:03:50.359 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.359 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.359 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:50.360 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.097477+0000", 2024-04-19T02:03:50.360 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014539899999999999 2024-04-19T02:03:50.360 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.360 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.360 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.360 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.099398+0000", 2024-04-19T02:03:50.360 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0019214150000000001 2024-04-19T02:03:50.360 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.360 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.360 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.360 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.360 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.360 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(osd.3.18:3 3.1s0 3:874b3480:::smithi05930959-36:head [assert-version v11,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:50.360 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.195845+0000", 2024-04-19T02:03:50.360 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 20.117134144000001, 2024-04-19T02:03:50.361 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012426504, 2024-04-19T02:03:50.361 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.361 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.361 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:50.361 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.361 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.361 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.361 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195845+0000", 2024-04-19T02:03:50.361 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.361 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.361 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.361 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.361 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195845+0000", 2024-04-19T02:03:50.361 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.361 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.362 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.362 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.362 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195847+0000", 2024-04-19T02:03:50.362 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0090000000000002e-06 2024-04-19T02:03:50.362 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.363 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.363 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.363 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195856+0000", 2024-04-19T02:03:50.363 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5650000000000002e-06 2024-04-19T02:03:50.363 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.363 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.363 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.363 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195871+0000", 2024-04-19T02:03:50.363 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5072000000000001e-05 2024-04-19T02:03:50.363 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.363 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.363 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.363 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195879+0000", 2024-04-19T02:03:50.363 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3680000000000002e-06 2024-04-19T02:03:50.364 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.364 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.364 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.364 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195978+0000", 2024-04-19T02:03:50.364 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.8202000000000006e-05 2024-04-19T02:03:50.364 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.364 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.364 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:50.364 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.196085+0000", 2024-04-19T02:03:50.364 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000107506 2024-04-19T02:03:50.364 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.364 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.364 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.364 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.208272+0000", 2024-04-19T02:03:50.364 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012186782 2024-04-19T02:03:50.365 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.365 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.365 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.365 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.365 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.365 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:447 3.1s0 3:88463a85:::smithi05930959-33:head [stat,delete] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:50.365 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.226277+0000", 2024-04-19T02:03:50.365 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 20.086702775999999, 2024-04-19T02:03:50.365 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.008527494, 2024-04-19T02:03:50.365 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.365 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.365 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:50.365 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:50.365 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:50.365 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:50.366 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 447 2024-04-19T02:03:50.366 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.366 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.366 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.366 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.366 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.226277+0000", 2024-04-19T02:03:50.366 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.366 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.366 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.366 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.366 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.226277+0000", 2024-04-19T02:03:50.366 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.366 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.367 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.367 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.367 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.226279+0000", 2024-04-19T02:03:50.367 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.526e-06 2024-04-19T02:03:50.367 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.367 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.367 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.367 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.226293+0000", 2024-04-19T02:03:50.367 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4093e-05 2024-04-19T02:03:50.367 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.367 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.367 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.367 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.226316+0000", 2024-04-19T02:03:50.368 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2875e-05 2024-04-19T02:03:50.368 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.368 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.368 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.368 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.226331+0000", 2024-04-19T02:03:50.368 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4677e-05 2024-04-19T02:03:50.368 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.368 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.368 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.368 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.226421+0000", 2024-04-19T02:03:50.368 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0027999999999999e-05 2024-04-19T02:03:50.369 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.369 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.369 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:50.369 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.226534+0000", 2024-04-19T02:03:50.369 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000113504 2024-04-19T02:03:50.369 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.369 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.369 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:50.369 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.227340+0000", 2024-04-19T02:03:50.369 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00080527699999999997 2024-04-19T02:03:50.369 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.369 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.369 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:50.369 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.228989+0000", 2024-04-19T02:03:50.369 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0016491940000000001 2024-04-19T02:03:50.370 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.370 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.370 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:50.370 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.234694+0000", 2024-04-19T02:03:50.370 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0057055960000000003 2024-04-19T02:03:50.370 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.370 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.370 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.370 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.234804+0000", 2024-04-19T02:03:50.370 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000109724 2024-04-19T02:03:50.370 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.370 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.370 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.370 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.370 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.371 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.bs1 47/26 ECSubWrite(tid=180, reqid=osd.3.0:180, at_version=47'36, trim_to=0'0, roll_forward_to=46'34))", 2024-04-19T02:03:50.371 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:31.080178+0000", 2024-04-19T02:03:50.371 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 19.232801365, 2024-04-19T02:03:50.371 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0072941200000000003, 2024-04-19T02:03:50.371 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.371 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.371 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:50.371 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.371 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.371 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.371 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.080178+0000", 2024-04-19T02:03:50.371 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.371 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.371 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.371 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.372 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.080178+0000", 2024-04-19T02:03:50.372 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.372 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.372 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.372 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.372 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.080179+0000", 2024-04-19T02:03:50.372 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.44e-07 2024-04-19T02:03:50.372 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.372 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.372 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.372 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.080216+0000", 2024-04-19T02:03:50.372 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.7452e-05 2024-04-19T02:03:50.372 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.372 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.372 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.373 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.080245+0000", 2024-04-19T02:03:50.373 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.9252999999999999e-05 2024-04-19T02:03:50.373 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.373 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.373 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.373 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.080277+0000", 2024-04-19T02:03:50.373 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2008999999999998e-05 2024-04-19T02:03:50.373 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.373 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.373 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.373 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.080366+0000", 2024-04-19T02:03:50.373 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.9118999999999998e-05 2024-04-19T02:03:50.373 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.373 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.374 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:50.374 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.080380+0000", 2024-04-19T02:03:50.374 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3580999999999999e-05 2024-04-19T02:03:50.374 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.374 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.374 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:50.374 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.087436+0000", 2024-04-19T02:03:50.374 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0070562699999999999 2024-04-19T02:03:50.374 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.374 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.374 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.374 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.087472+0000", 2024-04-19T02:03:50.374 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5892000000000001e-05 2024-04-19T02:03:50.374 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.375 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.375 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.375 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.375 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.375 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:465 3.1s0 3:874b3480:::smithi05930959-36:head [append 0~786432] snapc a=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:03:50.375 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:31.156851+0000", 2024-04-19T02:03:50.375 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 19.156128205999998, 2024-04-19T02:03:50.375 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0094295679999999993, 2024-04-19T02:03:50.375 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.375 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.375 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:50.375 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:50.375 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:50.376 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:50.376 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 465 2024-04-19T02:03:50.376 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.376 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.376 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.376 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.376 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.156851+0000", 2024-04-19T02:03:50.376 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.376 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.376 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.376 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.376 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.156851+0000", 2024-04-19T02:03:50.376 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.376 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.376 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.376 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.377 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.156854+0000", 2024-04-19T02:03:50.377 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8030000000000001e-06 2024-04-19T02:03:50.377 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.377 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.377 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.377 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.157331+0000", 2024-04-19T02:03:50.377 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00047709100000000002 2024-04-19T02:03:50.377 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.377 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.377 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.377 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.157357+0000", 2024-04-19T02:03:50.377 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6092e-05 2024-04-19T02:03:50.377 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.377 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.377 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.378 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.157371+0000", 2024-04-19T02:03:50.378 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.403e-05 2024-04-19T02:03:50.378 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.378 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.378 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.378 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.157471+0000", 2024-04-19T02:03:50.378 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000100049 2024-04-19T02:03:50.378 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.378 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.378 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:50.378 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.157607+0000", 2024-04-19T02:03:50.378 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013581500000000001 2024-04-19T02:03:50.378 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.379 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.379 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:50.379 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.159060+0000", 2024-04-19T02:03:50.379 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001452971 2024-04-19T02:03:50.379 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.379 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.379 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:50.379 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.160520+0000", 2024-04-19T02:03:50.379 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0014597449999999999 2024-04-19T02:03:50.379 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.379 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.379 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:50.379 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.166184+0000", 2024-04-19T02:03:50.379 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0056638799999999996 2024-04-19T02:03:50.380 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.380 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.380 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.380 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.166281+0000", 2024-04-19T02:03:50.380 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.7091999999999998e-05 2024-04-19T02:03:50.380 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.380 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.380 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.380 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.380 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.380 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:466 3.1s0 3:874b3480:::smithi05930959-36:head [append 0~491520] snapc a=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:03:50.380 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:31.160285+0000", 2024-04-19T02:03:50.380 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 19.152694415999999, 2024-04-19T02:03:50.380 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0066126589999999999, 2024-04-19T02:03:50.380 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.381 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.381 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:50.381 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:50.381 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:50.381 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:50.381 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 466 2024-04-19T02:03:50.381 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.381 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.381 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.381 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.381 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.160285+0000", 2024-04-19T02:03:50.381 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.381 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.381 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.382 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.382 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.160285+0000", 2024-04-19T02:03:50.382 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.382 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.382 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.382 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.382 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.160287+0000", 2024-04-19T02:03:50.382 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.857e-06 2024-04-19T02:03:50.382 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.382 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.382 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.382 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.160603+0000", 2024-04-19T02:03:50.382 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000316008 2024-04-19T02:03:50.382 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.382 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.383 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.383 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.160620+0000", 2024-04-19T02:03:50.383 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6977e-05 2024-04-19T02:03:50.383 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.383 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.383 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.383 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.160631+0000", 2024-04-19T02:03:50.383 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1403000000000001e-05 2024-04-19T02:03:50.383 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.383 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.383 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.383 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.160715+0000", 2024-04-19T02:03:50.383 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3820999999999996e-05 2024-04-19T02:03:50.383 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.384 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.384 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:50.384 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.160824+0000", 2024-04-19T02:03:50.384 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108744 2024-04-19T02:03:50.384 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.384 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.384 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:50.384 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.161896+0000", 2024-04-19T02:03:50.384 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001071916 2024-04-19T02:03:50.384 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.384 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.384 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:50.384 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.165411+0000", 2024-04-19T02:03:50.384 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0035147720000000002 2024-04-19T02:03:50.384 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.385 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.385 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:50.385 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.166822+0000", 2024-04-19T02:03:50.385 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0014117820000000001 2024-04-19T02:03:50.385 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.385 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.385 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.385 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.166898+0000", 2024-04-19T02:03:50.385 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.5378999999999998e-05 2024-04-19T02:03:50.385 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.385 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.385 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.385 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.385 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.386 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:469 3.1s0 3:874b3480:::smithi05930959-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:03:50.386 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:31.162590+0000", 2024-04-19T02:03:50.386 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 19.150389351000001, 2024-04-19T02:03:50.386 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0086854280000000002, 2024-04-19T02:03:50.386 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.386 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.386 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:50.386 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:50.386 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:50.386 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:50.386 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 469 2024-04-19T02:03:50.386 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.386 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.386 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.387 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.387 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162590+0000", 2024-04-19T02:03:50.387 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.387 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.387 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.387 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.387 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162590+0000", 2024-04-19T02:03:50.387 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.387 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.387 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.387 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.387 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162591+0000", 2024-04-19T02:03:50.387 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.1600000000000001e-07 2024-04-19T02:03:50.387 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.388 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.388 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.388 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162593+0000", 2024-04-19T02:03:50.388 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.458e-06 2024-04-19T02:03:50.388 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.388 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.388 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.388 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162604+0000", 2024-04-19T02:03:50.388 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0869e-05 2024-04-19T02:03:50.388 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.388 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.388 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.388 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162610+0000", 2024-04-19T02:03:50.388 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.2020000000000001e-06 2024-04-19T02:03:50.389 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.389 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.389 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.389 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.165692+0000", 2024-04-19T02:03:50.389 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0030820650000000002 2024-04-19T02:03:50.389 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.389 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.389 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:50.389 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.165778+0000", 2024-04-19T02:03:50.389 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5927000000000006e-05 2024-04-19T02:03:50.389 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.389 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.389 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.389 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.169491+0000", 2024-04-19T02:03:50.389 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0037127610000000002 2024-04-19T02:03:50.390 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.390 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.390 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:50.390 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.169588+0000", 2024-04-19T02:03:50.390 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.6850999999999998e-05 2024-04-19T02:03:50.390 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.390 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.390 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.390 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.171276+0000", 2024-04-19T02:03:50.390 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0016877789999999999 2024-04-19T02:03:50.390 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.390 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.390 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.390 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.390 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.391 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:646 3.1s0 3:8775257e:::smithi05930959-28:head [read 0~458752,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-19T02:03:50.391 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:46.185680+0000", 2024-04-19T02:03:50.391 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 4.1272991799999996, 2024-04-19T02:03:50.391 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0096160059999999999, 2024-04-19T02:03:50.391 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.391 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.391 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:50.391 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:50.391 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:50.391 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:50.391 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 646 2024-04-19T02:03:50.391 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.391 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.391 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.392 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.392 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185680+0000", 2024-04-19T02:03:50.392 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.392 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.392 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.392 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.392 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185680+0000", 2024-04-19T02:03:50.392 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.392 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.392 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.392 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.392 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185684+0000", 2024-04-19T02:03:50.392 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0799999999999999e-06 2024-04-19T02:03:50.392 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.392 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.393 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.393 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185707+0000", 2024-04-19T02:03:50.393 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2192e-05 2024-04-19T02:03:50.393 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.393 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.393 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.393 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185751+0000", 2024-04-19T02:03:50.393 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4098999999999999e-05 2024-04-19T02:03:50.393 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.393 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.393 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.393 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185770+0000", 2024-04-19T02:03:50.393 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9116000000000001e-05 2024-04-19T02:03:50.393 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.394 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.394 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.394 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185952+0000", 2024-04-19T02:03:50.394 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000181781 2024-04-19T02:03:50.394 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.394 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.394 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "op must wait for map", 2024-04-19T02:03:50.394 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185997+0000", 2024-04-19T02:03:50.394 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5802e-05 2024-04-19T02:03:50.394 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.394 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.394 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.394 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.189566+0000", 2024-04-19T02:03:50.394 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.003568323 2024-04-19T02:03:50.394 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.395 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.395 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:50.395 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.189763+0000", 2024-04-19T02:03:50.395 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00019720100000000001 2024-04-19T02:03:50.395 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.395 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.395 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.395 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.195296+0000", 2024-04-19T02:03:50.395 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0055334119999999997 2024-04-19T02:03:50.395 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.395 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.395 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.395 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.395 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.396 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:647 3.1s0 3:8775257e:::smithi05930959-28:head [read 0~458752,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-19T02:03:50.396 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:46.185813+0000", 2024-04-19T02:03:50.396 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 4.1271662869999997, 2024-04-19T02:03:50.396 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010132234, 2024-04-19T02:03:50.396 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.396 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.396 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:50.396 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:50.396 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:50.396 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:50.396 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 647 2024-04-19T02:03:50.396 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.396 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.396 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.396 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.397 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185813+0000", 2024-04-19T02:03:50.397 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.397 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.397 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.397 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.397 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185813+0000", 2024-04-19T02:03:50.397 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.397 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.397 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.397 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.397 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185815+0000", 2024-04-19T02:03:50.397 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4699999999999999e-06 2024-04-19T02:03:50.397 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.397 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.397 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.397 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185834+0000", 2024-04-19T02:03:50.398 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9494999999999999e-05 2024-04-19T02:03:50.398 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.398 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.398 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.398 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185855+0000", 2024-04-19T02:03:50.398 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1265999999999999e-05 2024-04-19T02:03:50.398 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.398 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.398 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.398 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185875+0000", 2024-04-19T02:03:50.398 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9772999999999999e-05 2024-04-19T02:03:50.398 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.398 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.398 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.399 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.186193+0000", 2024-04-19T02:03:50.399 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00031767699999999999 2024-04-19T02:03:50.399 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.399 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.399 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:03:50.399 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.186210+0000", 2024-04-19T02:03:50.399 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7306999999999999e-05 2024-04-19T02:03:50.399 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.399 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.399 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.399 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.191548+0000", 2024-04-19T02:03:50.399 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0053383149999999997 2024-04-19T02:03:50.399 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.399 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.399 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:50.399 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.191764+0000", 2024-04-19T02:03:50.400 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000215109 2024-04-19T02:03:50.400 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.400 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.400 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.400 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.195945+0000", 2024-04-19T02:03:50.400 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.004181822 2024-04-19T02:03:50.400 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.400 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.400 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.400 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.400 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.400 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:648 3.1s0 3:8775257e:::smithi05930959-28:head [read 0~458752,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-19T02:03:50.400 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:46.185903+0000", 2024-04-19T02:03:50.400 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 4.1270761900000004, 2024-04-19T02:03:50.400 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.011792667999999999, 2024-04-19T02:03:50.401 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.401 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.401 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:50.401 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:50.401 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:50.401 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:50.401 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 648 2024-04-19T02:03:50.401 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.401 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.401 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.401 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.401 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185903+0000", 2024-04-19T02:03:50.401 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.401 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.401 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.402 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.402 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185903+0000", 2024-04-19T02:03:50.402 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.402 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.402 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.402 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.402 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185905+0000", 2024-04-19T02:03:50.402 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.246e-06 2024-04-19T02:03:50.402 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.402 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.402 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.402 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185913+0000", 2024-04-19T02:03:50.402 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5199999999999997e-06 2024-04-19T02:03:50.402 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.403 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.403 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.403 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185957+0000", 2024-04-19T02:03:50.403 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4212000000000003e-05 2024-04-19T02:03:50.403 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.403 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.403 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.403 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185983+0000", 2024-04-19T02:03:50.403 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6118999999999998e-05 2024-04-19T02:03:50.403 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.403 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.403 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.403 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.186272+0000", 2024-04-19T02:03:50.403 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00028864 2024-04-19T02:03:50.404 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.404 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.404 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:03:50.404 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.186285+0000", 2024-04-19T02:03:50.404 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3446e-05 2024-04-19T02:03:50.404 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.404 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.404 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.404 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.193694+0000", 2024-04-19T02:03:50.404 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0074089300000000002 2024-04-19T02:03:50.404 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.404 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.404 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:50.404 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.193872+0000", 2024-04-19T02:03:50.404 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000177307 2024-04-19T02:03:50.405 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.405 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.405 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.405 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.197696+0000", 2024-04-19T02:03:50.405 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0038242480000000001 2024-04-19T02:03:50.405 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.405 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.405 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.405 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.405 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.405 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.ds7 66/26 ECSubWrite(tid=236, reqid=client.4509.0:743, at_version=66'79, trim_to=0'0, roll_forward_to=66'78))", 2024-04-19T02:03:50.405 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:50.118550+0000", 2024-04-19T02:03:50.405 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.19442906900000001, 2024-04-19T02:03:50.405 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0082560210000000005, 2024-04-19T02:03:50.405 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.406 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.406 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:50.406 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.406 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.406 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.406 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.118550+0000", 2024-04-19T02:03:50.406 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.406 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.406 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.406 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.406 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.118550+0000", 2024-04-19T02:03:50.406 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.406 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.406 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.407 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.407 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.118551+0000", 2024-04-19T02:03:50.407 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.4600000000000004e-07 2024-04-19T02:03:50.407 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.407 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.407 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.407 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.118786+0000", 2024-04-19T02:03:50.407 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00023447100000000001 2024-04-19T02:03:50.407 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.407 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.407 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.407 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.118803+0000", 2024-04-19T02:03:50.407 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7096999999999999e-05 2024-04-19T02:03:50.407 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.407 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.408 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.408 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.118814+0000", 2024-04-19T02:03:50.408 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1221e-05 2024-04-19T02:03:50.408 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.408 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.408 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.408 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.118888+0000", 2024-04-19T02:03:50.408 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.4293000000000002e-05 2024-04-19T02:03:50.408 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.408 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.408 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:50.408 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.118900+0000", 2024-04-19T02:03:50.408 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1384e-05 2024-04-19T02:03:50.408 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.409 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.409 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:50.409 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.126744+0000", 2024-04-19T02:03:50.409 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0078441550000000002 2024-04-19T02:03:50.409 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.409 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.409 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.409 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.126806+0000", 2024-04-19T02:03:50.409 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.2654e-05 2024-04-19T02:03:50.409 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.409 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.409 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.409 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.409 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.409 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:50.411 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-19T02:03:50.414 INFO:tasks.rados.rados.0.smithi059.stdout:285: finishing write tid 1 to smithi05930959-16 2024-04-19T02:03:50.414 INFO:tasks.rados.rados.0.smithi059.stdout:285: finishing write tid 2 to smithi05930959-16 2024-04-19T02:03:50.414 INFO:tasks.rados.rados.0.smithi059.stdout:278: done (7 left) 2024-04-19T02:03:50.414 INFO:tasks.rados.rados.0.smithi059.stdout:279: done (6 left) 2024-04-19T02:03:50.414 INFO:tasks.rados.rados.0.smithi059.stdout:281: done (5 left) 2024-04-19T02:03:50.414 INFO:tasks.rados.rados.0.smithi059.stdout:282: done (4 left) 2024-04-19T02:03:50.414 INFO:tasks.rados.rados.0.smithi059.stdout:283: done (3 left) 2024-04-19T02:03:50.414 INFO:tasks.rados.rados.0.smithi059.stdout:284: done (2 left) 2024-04-19T02:03:50.414 INFO:tasks.rados.rados.0.smithi059.stdout:286: done (1 left) 2024-04-19T02:03:50.415 INFO:tasks.rados.rados.0.smithi059.stdout:287: copy_from oid 15 from oid 33 current snap is 21 2024-04-19T02:03:50.415 INFO:tasks.rados.rados.0.smithi059.stdout:285: finishing write tid 3 to smithi05930959-16 2024-04-19T02:03:50.415 INFO:tasks.rados.rados.0.smithi059.stdout:285: finishing write tid 4 to smithi05930959-16 2024-04-19T02:03:50.415 INFO:tasks.rados.rados.0.smithi059.stdout:285: finishing write tid 5 to smithi05930959-16 2024-04-19T02:03:50.415 INFO:tasks.rados.rados.0.smithi059.stdout:285: finishing write tid 6 to smithi05930959-16 2024-04-19T02:03:50.415 INFO:tasks.rados.rados.0.smithi059.stdout:285: oid 16 updating version 0 to 78 2024-04-19T02:03:50.415 INFO:tasks.rados.rados.0.smithi059.stdout:285: oid 16 version 78 is already newer than 77 2024-04-19T02:03:50.415 INFO:tasks.rados.rados.0.smithi059.stdout:285: oid 16 updating version 78 to 79 2024-04-19T02:03:50.415 INFO:tasks.rados.rados.0.smithi059.stdout:285: oid 16 updating version 79 to 80 2024-04-19T02:03:50.415 INFO:tasks.rados.rados.0.smithi059.stdout:285: oid 16 updating version 80 to 81 2024-04-19T02:03:50.415 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 16 v 81 (ObjNum 134 snap 21 seq_num 134) dirty exists 2024-04-19T02:03:50.415 INFO:tasks.rados.rados.0.smithi059.stdout:285: left oid 16 (ObjNum 134 snap 21 seq_num 134) 2024-04-19T02:03:50.415 INFO:tasks.rados.rados.0.smithi059.stdout:285: done (1 left) 2024-04-19T02:03:50.415 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 18 current snap is 21 2024-04-19T02:03:50.415 INFO:tasks.rados.rados.0.smithi059.stdout:288: seq_num 136 ranges {1490944=737280,2228224=770048,2998272=540672,3538944=245760} 2024-04-19T02:03:50.416 INFO:tasks.rados.rados.0.smithi059.stdout:288: writing smithi05930959-18 from 1490944 to 2228224 tid 1 2024-04-19T02:03:50.416 INFO:tasks.rados.rados.0.smithi059.stdout:288: writing smithi05930959-18 from 2228224 to 2998272 tid 2 2024-04-19T02:03:50.416 INFO:tasks.rados.rados.0.smithi059.stdout:288: writing smithi05930959-18 from 2998272 to 3538944 tid 3 2024-04-19T02:03:50.416 INFO:tasks.rados.rados.0.smithi059.stdout:288: writing smithi05930959-18 from 3538944 to 3784704 tid 4 2024-04-19T02:03:50.416 INFO:tasks.rados.rados.0.smithi059.stdout:289: copy_from oid 31 from oid 2 current snap is 21 2024-04-19T02:03:50.416 INFO:tasks.rados.rados.0.smithi059.stdout:287: finishing copy_from to smithi05930959-15 2024-04-19T02:03:50.416 INFO:tasks.rados.rados.0.smithi059.stdout:287: got expected ENOENT (src dne) 2024-04-19T02:03:50.416 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 6 current snap is 21 2024-04-19T02:03:50.416 INFO:tasks.rados.rados.0.smithi059.stdout:290: seq_num 137 ranges {98304=753664,851968=786432,1638400=655360,2293760=114688} 2024-04-19T02:03:50.416 INFO:tasks.rados.rados.0.smithi059.stdout:290: writing smithi05930959-6 from 98304 to 851968 tid 1 2024-04-19T02:03:50.416 INFO:tasks.rados.rados.0.smithi059.stdout:290: writing smithi05930959-6 from 851968 to 1638400 tid 2 2024-04-19T02:03:50.416 INFO:tasks.rados.rados.0.smithi059.stdout:290: writing smithi05930959-6 from 1638400 to 2293760 tid 3 2024-04-19T02:03:50.416 INFO:tasks.rados.rados.0.smithi059.stdout:290: writing smithi05930959-6 from 2293760 to 2408448 tid 4 2024-04-19T02:03:50.416 INFO:tasks.rados.rados.0.smithi059.stdout:291: snap_create 2024-04-19T02:03:50.416 INFO:tasks.rados.rados.0.smithi059.stdout:287: finishing copy_from racing read to smithi05930959-15 2024-04-19T02:03:50.417 INFO:tasks.rados.rados.0.smithi059.stdout:288: finishing write tid 1 to smithi05930959-18 2024-04-19T02:03:50.417 INFO:tasks.rados.rados.0.smithi059.stdout:289: finishing copy_from racing read to smithi05930959-31 2024-04-19T02:03:50.417 INFO:tasks.rados.rados.0.smithi059.stdout:289: finishing copy_from to smithi05930959-31 2024-04-19T02:03:50.417 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 31 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-19T02:03:50.417 INFO:tasks.rados.rados.0.smithi059.stdout:288: finishing write tid 2 to smithi05930959-18 2024-04-19T02:03:50.417 INFO:tasks.rados.rados.0.smithi059.stdout:288: finishing write tid 3 to smithi05930959-18 2024-04-19T02:03:50.417 INFO:tasks.rados.rados.0.smithi059.stdout:288: finishing write tid 4 to smithi05930959-18 2024-04-19T02:03:50.417 INFO:tasks.rados.rados.0.smithi059.stdout:288: finishing write tid 5 to smithi05930959-18 2024-04-19T02:03:50.417 INFO:tasks.rados.rados.0.smithi059.stdout:288: finishing write tid 6 to smithi05930959-18 2024-04-19T02:03:50.417 INFO:tasks.rados.rados.0.smithi059.stdout:288: oid 18 updating version 0 to 73 2024-04-19T02:03:50.417 INFO:tasks.rados.rados.0.smithi059.stdout:288: oid 18 version 73 is already newer than 69 2024-04-19T02:03:50.417 INFO:tasks.rados.rados.0.smithi059.stdout:288: oid 18 version 73 is already newer than 72 2024-04-19T02:03:50.417 INFO:tasks.rados.rados.0.smithi059.stdout:288: oid 18 version 73 is already newer than 70 2024-04-19T02:03:50.417 INFO:tasks.rados.rados.0.smithi059.stdout:288: oid 18 version 73 is already newer than 71 2024-04-19T02:03:50.418 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 18 v 73 (ObjNum 136 snap 21 seq_num 136) dirty exists 2024-04-19T02:03:50.418 INFO:tasks.rados.rados.0.smithi059.stdout:288: left oid 18 (ObjNum 136 snap 21 seq_num 136) 2024-04-19T02:03:50.418 INFO:tasks.rados.rados.0.smithi059.stdout:290: finishing write tid 1 to smithi05930959-6 2024-04-19T02:03:50.418 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:50.410+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:50.419 INFO:tasks.rados.rados.0.smithi059.stdout:290: finishing write tid 2 to smithi05930959-6 2024-04-19T02:03:50.419 INFO:tasks.rados.rados.0.smithi059.stdout:290: finishing write tid 3 to smithi05930959-6 2024-04-19T02:03:50.419 INFO:tasks.rados.rados.0.smithi059.stdout:290: finishing write tid 4 to smithi05930959-6 2024-04-19T02:03:50.419 INFO:tasks.rados.rados.0.smithi059.stdout:290: finishing write tid 5 to smithi05930959-6 2024-04-19T02:03:50.420 INFO:tasks.rados.rados.0.smithi059.stdout:290: finishing write tid 6 to smithi05930959-6 2024-04-19T02:03:50.420 INFO:tasks.rados.rados.0.smithi059.stdout:290: oid 6 updating version 0 to 67 2024-04-19T02:03:50.420 INFO:tasks.rados.rados.0.smithi059.stdout:290: oid 6 version 67 is already newer than 65 2024-04-19T02:03:50.420 INFO:tasks.rados.rados.0.smithi059.stdout:290: oid 6 updating version 67 to 68 2024-04-19T02:03:50.420 INFO:tasks.rados.rados.0.smithi059.stdout:290: oid 6 version 68 is already newer than 64 2024-04-19T02:03:50.420 INFO:tasks.rados.rados.0.smithi059.stdout:290: oid 6 version 68 is already newer than 66 2024-04-19T02:03:50.421 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 6 v 68 (ObjNum 137 snap 21 seq_num 137) dirty exists 2024-04-19T02:03:50.421 INFO:tasks.rados.rados.0.smithi059.stdout:290: left oid 6 (ObjNum 137 snap 21 seq_num 137) 2024-04-19T02:03:50.500 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-19T02:03:50.500 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-19T02:03:50.500 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-04-19T02:03:50.501 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T02:03:50.537 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:50.538 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:03:50.538 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-04-19T02:03:50.538 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:50.538 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:50.538+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:50.553 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-19T02:03:50.643 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:50.646+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:50.714 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:50.717+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:50.717 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:50.718 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:03:50.718 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-04-19T02:03:50.718 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-04-19T02:03:50.718 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:50.732 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-19T02:03:50.806 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:03:50.807 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-19T02:03:50.815 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:50.818+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:50.825 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2024-04-19T02:03:50.218484+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98157088,"num_objects":83,"num_object_clones":35,"num_object_copies":652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":12,"num_read":470,"num_read_kb":123150,"num_write":764,"num_write_kb":123466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":924,"ondisk_log_size":924,"up":146,"acting":146,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1132462080,"kb_used":553952,"kb_used_data":196384,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1131908128,"statfs":{"total":1159641169920,"available":1159073923072,"internally_reserved":0,"allocated":201097216,"data_stored":247717996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12681216,"num_objects":13,"num_object_clones":12,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":6,"num_read":24,"num_read_kb":15228,"num_write":56,"num_write_kb":18384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001299"},"pg_stats":[{"pgid":"3.f","version":"60'43","reported_seq":244,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-19T02:03:48.061054+0000","last_change":"2024-04-19T02:03:48.061980+0000","last_active":"2024-04-19T02:03:48.061054+0000","last_peered":"2024-04-19T02:03:48.061054+0000","last_clean":"2024-04-19T02:03:48.061054+0000","last_became_active":"2024-04-19T02:03:11.728136+0000","last_became_peered":"2024-04-19T02:03:11.728136+0000","last_unstale":"2024-04-19T02:03:48.061054+0000","last_undegraded":"2024-04-19T02:03:48.061054+0000","last_fullsized":"2024-04-19T02:03:48.061054+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:31.275514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001702734,"stat_sum":{"num_bytes":8192000,"num_objects":4,"num_object_clones":1,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":10858,"num_write":27,"num_write_kb":9008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,2,3,5,8,0,7],"acting":[11,4,2,3,5,8,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"2"}]},{"pgid":"3.e","version":"64'50","reported_seq":252,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-19T02:03:48.080826+0000","last_change":"2024-04-19T02:03:48.082694+0000","last_active":"2024-04-19T02:03:48.080826+0000","last_peered":"2024-04-19T02:03:48.080826+0000","last_clean":"2024-04-19T02:03:48.080826+0000","last_became_active":"2024-04-19T02:03:11.730135+0000","last_became_peered":"2024-04-19T02:03:11.730135+0000","last_unstale":"2024-04-19T02:03:48.080826+0000","last_undegraded":"2024-04-19T02:03:48.080826+0000","last_fullsized":"2024-04-19T02:03:48.080826+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:22.794432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018538466999999999,"stat_sum":{"num_bytes":6586368,"num_objects":4,"num_object_clones":1,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":7016,"num_write":25,"num_write_kb":7648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5,11,0,2,3,9],"acting":[7,4,5,11,0,2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"2"}]},{"pgid":"3.d","version":"64'75","reported_seq":278,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-19T02:03:49.055947+0000","last_change":"2024-04-19T02:03:48.084532+0000","last_active":"2024-04-19T02:03:49.055947+0000","last_peered":"2024-04-19T02:03:49.055947+0000","last_clean":"2024-04-19T02:03:49.055947+0000","last_became_active":"2024-04-19T02:03:11.632682+0000","last_became_peered":"2024-04-19T02:03:11.632682+0000","last_unstale":"2024-04-19T02:03:49.055947+0000","last_undegraded":"2024-04-19T02:03:49.055947+0000","last_fullsized":"2024-04-19T02:03:49.055947+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:26.215417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018311895000000002,"stat_sum":{"num_bytes":10010624,"num_objects":9,"num_object_clones":5,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":28,"num_read_kb":5819,"num_write":53,"num_write_kb":12208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7,4,5,3,9,0],"acting":[8,11,7,4,5,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"2"}]},{"pgid":"3.c","version":"64'62","reported_seq":278,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-19T02:03:48.069613+0000","last_change":"2024-04-19T02:03:48.071316+0000","last_active":"2024-04-19T02:03:48.069613+0000","last_peered":"2024-04-19T02:03:48.069613+0000","last_clean":"2024-04-19T02:03:48.069613+0000","last_became_active":"2024-04-19T02:03:11.731568+0000","last_became_peered":"2024-04-19T02:03:11.731568+0000","last_unstale":"2024-04-19T02:03:48.069613+0000","last_undegraded":"2024-04-19T02:03:48.069613+0000","last_fullsized":"2024-04-19T02:03:48.069613+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:33.416097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017738041999999999,"stat_sum":{"num_bytes":7323648,"num_objects":8,"num_object_clones":4,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":47,"num_read_kb":16192,"num_write":32,"num_write_kb":10704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,11,7,4,2,0,5],"acting":[9,3,11,7,4,2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"2"}]},{"pgid":"3.b","version":"64'60","reported_seq":265,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-19T02:03:49.055677+0000","last_change":"2024-04-19T02:03:48.087466+0000","last_active":"2024-04-19T02:03:49.055677+0000","last_peered":"2024-04-19T02:03:49.055677+0000","last_clean":"2024-04-19T02:03:49.055677+0000","last_became_active":"2024-04-19T02:03:11.677316+0000","last_became_peered":"2024-04-19T02:03:11.677316+0000","last_unstale":"2024-04-19T02:03:49.055677+0000","last_undegraded":"2024-04-19T02:03:49.055677+0000","last_fullsized":"2024-04-19T02:03:49.055677+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:17.190920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.031615879999999999,"stat_sum":{"num_bytes":4947968,"num_objects":4,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":32,"num_read_kb":13425,"num_write":29,"num_write_kb":9120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8,6,4,10,1,7],"acting":[3,0,8,6,4,10,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"2"}]},{"pgid":"3.a","version":"60'73","reported_seq":273,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-19T02:03:48.060354+0000","last_change":"2024-04-19T02:03:48.062547+0000","last_active":"2024-04-19T02:03:48.060354+0000","last_peered":"2024-04-19T02:03:48.060354+0000","last_clean":"2024-04-19T02:03:48.060354+0000","last_became_active":"2024-04-19T02:03:11.727996+0000","last_became_peered":"2024-04-19T02:03:11.727996+0000","last_unstale":"2024-04-19T02:03:48.060354+0000","last_undegraded":"2024-04-19T02:03:48.060354+0000","last_fullsized":"2024-04-19T02:03:48.060354+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:34.471668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025020099999999998,"stat_sum":{"num_bytes":11534336,"num_objects":7,"num_object_clones":3,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":30,"num_read_kb":25542,"num_write":85,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,1,9,5,7,3,2],"acting":[11,8,1,9,5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"2"}]},{"pgid":"3.9","version":"64'45","reported_seq":245,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-19T02:03:49.055128+0000","last_change":"2024-04-19T02:03:48.079314+0000","last_active":"2024-04-19T02:03:49.055128+0000","last_peered":"2024-04-19T02:03:49.055128+0000","last_clean":"2024-04-19T02:03:49.055128+0000","last_became_active":"2024-04-19T02:03:11.729255+0000","last_became_peered":"2024-04-19T02:03:11.729255+0000","last_unstale":"2024-04-19T02:03:49.055128+0000","last_undegraded":"2024-04-19T02:03:49.055128+0000","last_fullsized":"2024-04-19T02:03:49.055128+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:33.989189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.025632599999999998,"stat_sum":{"num_bytes":6012928,"num_objects":6,"num_object_clones":3,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":17,"num_read_kb":4617,"num_write":30,"num_write_kb":8576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,10,2,9,1,7,11],"acting":[4,3,10,2,9,1,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"2"}]},{"pgid":"3.8","version":"57'63","reported_seq":270,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-19T02:03:49.055096+0000","last_change":"2024-04-19T02:03:48.055291+0000","last_active":"2024-04-19T02:03:49.055096+0000","last_peered":"2024-04-19T02:03:49.055096+0000","last_clean":"2024-04-19T02:03:49.055096+0000","last_became_active":"2024-04-19T02:03:11.679384+0000","last_became_peered":"2024-04-19T02:03:11.679384+0000","last_unstale":"2024-04-19T02:03:49.055096+0000","last_undegraded":"2024-04-19T02:03:49.055096+0000","last_fullsized":"2024-04-19T02:03:49.055096+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:28.129843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015500749999999999,"stat_sum":{"num_bytes":9764864,"num_objects":6,"num_object_clones":3,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":14509,"num_write":32,"num_write_kb":9680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7,11,5,9,6,10],"acting":[4,1,7,11,5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"2"}]},{"pgid":"1.0","version":"21'137","reported_seq":275,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-19T02:03:49.055215+0000","last_change":"2024-04-19T02:02:59.974166+0000","last_active":"2024-04-19T02:03:49.055215+0000","last_peered":"2024-04-19T02:03:49.055215+0000","last_clean":"2024-04-19T02:03:49.055215+0000","last_became_active":"2024-04-19T02:02:59.972005+0000","last_became_peered":"2024-04-19T02:02:59.972005+0000","last_unstale":"2024-04-19T02:03:49.055215+0000","last_undegraded":"2024-04-19T02:03:49.055215+0000","last_fullsized":"2024-04-19T02:03:49.055215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:02:58.486225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:02:58.486225+0000","last_clean_scrub_stamp":"2024-04-19T02:02:58.486225+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:20.275707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":94,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-19T02:03:48.060292+0000","last_change":"2024-04-19T02:03:03.026806+0000","last_active":"2024-04-19T02:03:48.060292+0000","last_peered":"2024-04-19T02:03:48.060292+0000","last_clean":"2024-04-19T02:03:48.060292+0000","last_became_active":"2024-04-19T02:03:03.026063+0000","last_became_peered":"2024-04-19T02:03:03.026063+0000","last_unstale":"2024-04-19T02:03:48.060292+0000","last_undegraded":"2024-04-19T02:03:48.060292+0000","last_fullsized":"2024-04-19T02:03:48.060292+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:19.673016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"64'133","reported_seq":345,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-19T02:03:49.056034+0000","last_change":"2024-04-19T02:03:48.095791+0000","last_active":"2024-04-19T02:03:49.056034+0000","last_peered":"2024-04-19T02:03:49.056034+0000","last_clean":"2024-04-19T02:03:49.056034+0000","last_became_active":"2024-04-19T02:03:11.630475+0000","last_became_peered":"2024-04-19T02:03:11.630475+0000","last_unstale":"2024-04-19T02:03:49.056034+0000","last_undegraded":"2024-04-19T02:03:49.056034+0000","last_fullsized":"2024-04-19T02:03:49.056034+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:35.277960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.037896317999999998,"stat_sum":{"num_bytes":12533760,"num_objects":11,"num_object_clones":4,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":53,"num_read_kb":14364,"num_write":101,"num_write_kb":18064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10,4,8,1,11,9],"acting":[3,5,10,4,8,1,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":96,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-19T02:03:49.055195+0000","last_change":"2024-04-19T02:03:02.561076+0000","last_active":"2024-04-19T02:03:49.055195+0000","last_peered":"2024-04-19T02:03:49.055195+0000","last_clean":"2024-04-19T02:03:49.055195+0000","last_became_active":"2024-04-19T02:03:02.560210+0000","last_became_peered":"2024-04-19T02:03:02.560210+0000","last_unstale":"2024-04-19T02:03:49.055195+0000","last_undegraded":"2024-04-19T02:03:49.055195+0000","last_fullsized":"2024-04-19T02:03:49.055195+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:16.900004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"47'21","reported_seq":202,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-19T02:03:45.172859+0000","last_change":"2024-04-19T02:03:44.171725+0000","last_active":"2024-04-19T02:03:45.172859+0000","last_peered":"2024-04-19T02:03:45.172859+0000","last_clean":"2024-04-19T02:03:45.172859+0000","last_became_active":"2024-04-19T02:03:11.633394+0000","last_became_peered":"2024-04-19T02:03:11.633394+0000","last_unstale":"2024-04-19T02:03:45.172859+0000","last_undegraded":"2024-04-19T02:03:45.172859+0000","last_fullsized":"2024-04-19T02:03:45.172859+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:18.515114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010184479999999999,"stat_sum":{"num_bytes":4096000,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":1350,"num_write":15,"num_write_kb":4528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,4,5,10,7,9,1],"acting":[0,11,4,5,10,7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b","length":"1"},{"start":"e","length":"2"},{"start":"11","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":91,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-19T02:03:49.055462+0000","last_change":"2024-04-19T02:03:09.594947+0000","last_active":"2024-04-19T02:03:49.055462+0000","last_peered":"2024-04-19T02:03:49.055462+0000","last_clean":"2024-04-19T02:03:49.055462+0000","last_became_active":"2024-04-19T02:03:09.594072+0000","last_became_peered":"2024-04-19T02:03:09.594072+0000","last_unstale":"2024-04-19T02:03:49.055462+0000","last_undegraded":"2024-04-19T02:03:49.055462+0000","last_fullsized":"2024-04-19T02:03:49.055462+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:29.461650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":190,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-19T02:03:48.062171+0000","last_change":"2024-04-19T02:03:48.063105+0000","last_active":"2024-04-19T02:03:48.062171+0000","last_peered":"2024-04-19T02:03:48.062171+0000","last_clean":"2024-04-19T02:03:48.062171+0000","last_became_active":"2024-04-19T02:03:11.678078+0000","last_became_peered":"2024-04-19T02:03:11.678078+0000","last_unstale":"2024-04-19T02:03:48.062171+0000","last_undegraded":"2024-04-19T02:03:48.062171+0000","last_fullsized":"2024-04-19T02:03:48.062171+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:35.179916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022953399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,9,1,6,4,0,10],"acting":[5,7,9,1,6,4,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"2"}]},{"pgid":"2.2","version":"0'0","reported_seq":96,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-19T02:03:49.055538+0000","last_change":"2024-04-19T02:03:02.560133+0000","last_active":"2024-04-19T02:03:49.055538+0000","last_peered":"2024-04-19T02:03:49.055538+0000","last_clean":"2024-04-19T02:03:49.055538+0000","last_became_active":"2024-04-19T02:03:02.559377+0000","last_became_peered":"2024-04-19T02:03:02.559377+0000","last_unstale":"2024-04-19T02:03:49.055538+0000","last_undegraded":"2024-04-19T02:03:49.055538+0000","last_fullsized":"2024-04-19T02:03:49.055538+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:30.728400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"57'24","reported_seq":218,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-19T02:03:48.070968+0000","last_change":"2024-04-19T02:03:48.071838+0000","last_active":"2024-04-19T02:03:48.070968+0000","last_peered":"2024-04-19T02:03:48.070968+0000","last_clean":"2024-04-19T02:03:48.070968+0000","last_became_active":"2024-04-19T02:03:11.677099+0000","last_became_peered":"2024-04-19T02:03:11.677099+0000","last_unstale":"2024-04-19T02:03:48.070968+0000","last_undegraded":"2024-04-19T02:03:48.070968+0000","last_fullsized":"2024-04-19T02:03:48.070968+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:24.597314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014152290000000001,"stat_sum":{"num_bytes":4276224,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":2369,"num_write":53,"num_write_kb":4176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,0,3,4,7,9,6],"acting":[10,8,0,3,4,7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"2"}]},{"pgid":"3.5","version":"64'39","reported_seq":230,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-19T02:03:48.087302+0000","last_change":"2024-04-19T02:03:48.089252+0000","last_active":"2024-04-19T02:03:48.087302+0000","last_peered":"2024-04-19T02:03:48.087302+0000","last_clean":"2024-04-19T02:03:48.087302+0000","last_became_active":"2024-04-19T02:03:11.728949+0000","last_became_peered":"2024-04-19T02:03:11.728949+0000","last_unstale":"2024-04-19T02:03:48.087302+0000","last_undegraded":"2024-04-19T02:03:48.087302+0000","last_fullsized":"2024-04-19T02:03:48.087302+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:35.053744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019649005000000001,"stat_sum":{"num_bytes":2293760,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":8,"num_read_kb":5,"num_write":37,"num_write_kb":3584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2,9,8,1,6,5],"acting":[10,3,2,9,8,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":94,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-19T02:03:48.062150+0000","last_change":"2024-04-19T02:03:02.597582+0000","last_active":"2024-04-19T02:03:48.062150+0000","last_peered":"2024-04-19T02:03:48.062150+0000","last_clean":"2024-04-19T02:03:48.062150+0000","last_became_active":"2024-04-19T02:03:02.597118+0000","last_became_peered":"2024-04-19T02:03:02.597118+0000","last_unstale":"2024-04-19T02:03:48.062150+0000","last_undegraded":"2024-04-19T02:03:48.062150+0000","last_fullsized":"2024-04-19T02:03:48.062150+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:13.683143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"64'67","reported_seq":263,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-19T02:03:48.081818+0000","last_change":"2024-04-19T02:03:48.084395+0000","last_active":"2024-04-19T02:03:48.081818+0000","last_peered":"2024-04-19T02:03:48.081818+0000","last_clean":"2024-04-19T02:03:48.081818+0000","last_became_active":"2024-04-19T02:03:11.730165+0000","last_became_peered":"2024-04-19T02:03:11.730165+0000","last_unstale":"2024-04-19T02:03:48.081818+0000","last_undegraded":"2024-04-19T02:03:48.081818+0000","last_fullsized":"2024-04-19T02:03:48.081818+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:25.267271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022256564,"stat_sum":{"num_bytes":7651328,"num_objects":9,"num_object_clones":4,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":25,"num_read_kb":6443,"num_write":52,"num_write_kb":9296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,10,8,7,2,11,0],"acting":[1,5,10,8,7,2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"2"}]},{"pgid":"2.5","version":"0'0","reported_seq":94,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-19T02:03:48.064138+0000","last_change":"2024-04-19T02:03:02.595929+0000","last_active":"2024-04-19T02:03:48.064138+0000","last_peered":"2024-04-19T02:03:48.064138+0000","last_clean":"2024-04-19T02:03:48.064138+0000","last_became_active":"2024-04-19T02:03:02.595232+0000","last_became_peered":"2024-04-19T02:03:02.595232+0000","last_unstale":"2024-04-19T02:03:48.064138+0000","last_undegraded":"2024-04-19T02:03:48.064138+0000","last_fullsized":"2024-04-19T02:03:48.064138+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:04.077370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"64'26","reported_seq":218,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-19T02:03:49.056038+0000","last_change":"2024-04-19T02:03:48.078738+0000","last_active":"2024-04-19T02:03:49.056038+0000","last_peered":"2024-04-19T02:03:49.056038+0000","last_clean":"2024-04-19T02:03:49.056038+0000","last_became_active":"2024-04-19T02:03:11.675056+0000","last_became_peered":"2024-04-19T02:03:11.675056+0000","last_unstale":"2024-04-19T02:03:49.056038+0000","last_undegraded":"2024-04-19T02:03:49.056038+0000","last_fullsized":"2024-04-19T02:03:49.056038+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:32.848250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.021654606,"stat_sum":{"num_bytes":2162688,"num_objects":3,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":5,"num_read_kb":2,"num_write":9,"num_write_kb":2496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,9,0,7,5,11,1],"acting":[3,6,9,0,7,5,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":94,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-19T02:03:48.062806+0000","last_change":"2024-04-19T02:03:02.988302+0000","last_active":"2024-04-19T02:03:48.062806+0000","last_peered":"2024-04-19T02:03:48.062806+0000","last_clean":"2024-04-19T02:03:48.062806+0000","last_became_active":"2024-04-19T02:03:02.988073+0000","last_became_peered":"2024-04-19T02:03:02.988073+0000","last_unstale":"2024-04-19T02:03:48.062806+0000","last_undegraded":"2024-04-19T02:03:48.062806+0000","last_fullsized":"2024-04-19T02:03:48.062806+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:28.455271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"57'6","reported_seq":182,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-19T02:03:45.173700+0000","last_change":"2024-04-19T02:03:44.171995+0000","last_active":"2024-04-19T02:03:45.173700+0000","last_peered":"2024-04-19T02:03:45.173700+0000","last_clean":"2024-04-19T02:03:45.173700+0000","last_became_active":"2024-04-19T02:03:11.673940+0000","last_became_peered":"2024-04-19T02:03:11.673940+0000","last_unstale":"2024-04-19T02:03:45.173700+0000","last_undegraded":"2024-04-19T02:03:45.173700+0000","last_fullsized":"2024-04-19T02:03:45.173700+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:18.325069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058315400000000001,"stat_sum":{"num_bytes":180224,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":6,"num_read_kb":530,"num_write":3,"num_write_kb":176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9,8,4,5,6,10],"acting":[0,1,9,8,4,5,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b","length":"1"},{"start":"e","length":"2"},{"start":"11","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":96,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-19T02:03:49.055914+0000","last_change":"2024-04-19T02:03:02.570839+0000","last_active":"2024-04-19T02:03:49.055914+0000","last_peered":"2024-04-19T02:03:49.055914+0000","last_clean":"2024-04-19T02:03:49.055914+0000","last_became_active":"2024-04-19T02:03:02.570116+0000","last_became_peered":"2024-04-19T02:03:02.570116+0000","last_unstale":"2024-04-19T02:03:49.055914+0000","last_undegraded":"2024-04-19T02:03:49.055914+0000","last_fullsized":"2024-04-19T02:03:49.055914+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:23.134425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":97566720,"num_objects":81,"num_object_clones":35,"num_object_copies":648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":12,"num_read":344,"num_read_kb":123041,"num_write":583,"num_write_kb":120528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":191934464,"data_stored":242511872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":787,"ondisk_log_size":787,"up":128,"acting":128,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":19,"seq":81604378637,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42392,"kb_used_data":11212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329448,"statfs":{"total":96636764160,"available":96593354752,"internally_reserved":0,"allocated":11481088,"data_stored":14413859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":19,"seq":81604378636,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43328,"kb_used_data":12144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328512,"statfs":{"total":96636764160,"available":96592396288,"internally_reserved":0,"allocated":12435456,"data_stored":14684420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411341,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52860,"kb_used_data":21672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94318980,"statfs":{"total":96636764160,"available":96582635520,"internally_reserved":0,"allocated":22192128,"data_stored":27358319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411342,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50640,"kb_used_data":19452,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321200,"statfs":{"total":96636764160,"available":96584908800,"internally_reserved":0,"allocated":19918848,"data_stored":24913007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411342,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47472,"kb_used_data":16284,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324368,"statfs":{"total":96636764160,"available":96588152832,"internally_reserved":0,"allocated":16674816,"data_stored":20793200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411342,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47888,"kb_used_data":20864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94323952,"statfs":{"total":96636764160,"available":96587726848,"internally_reserved":0,"allocated":21364736,"data_stored":27128943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":18,"seq":77309411340,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33500,"kb_used_data":6476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338340,"statfs":{"total":96636764160,"available":96602460160,"internally_reserved":0,"allocated":6631424,"data_stored":8340591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411341,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45780,"kb_used_data":18756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94326060,"statfs":{"total":96636764160,"available":96589885440,"internally_reserved":0,"allocated":19206144,"data_stored":22717551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":18,"seq":77309411341,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50416,"kb_used_data":19232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321424,"statfs":{"total":96636764160,"available":96585138176,"internally_reserved":0,"allocated":19693568,"data_stored":25105519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":17,"seq":73014444045,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40352,"kb_used_data":13332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331488,"statfs":{"total":96636764160,"available":96595443712,"internally_reserved":0,"allocated":13651968,"data_stored":15717206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":17,"seq":73014444045,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52508,"kb_used_data":21328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94319332,"statfs":{"total":96636764160,"available":96582995968,"internally_reserved":0,"allocated":21839872,"data_stored":26711695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":17,"seq":73014444045,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46816,"kb_used_data":15632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325024,"statfs":{"total":96636764160,"available":96588824576,"internally_reserved":0,"allocated":16007168,"data_stored":19833686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11747328,"data_stored":14278656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":15294464,"data_stored":19501056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11309056,"data_stored":14794752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":18968576,"data_stored":24674304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18481152,"data_stored":22286336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20582400,"data_stored":25751552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5906432,"data_stored":7909376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20639744,"data_stored":26697728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":15405056,"data_stored":19824640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":19193856,"data_stored":24481792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":12939264,"data_stored":15384576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":21467136,"data_stored":26927104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T02:03:50.827 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pgp_num 16 2024-04-19T02:03:50.902 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:50.902 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-04-19T02:03:50.902 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-04-19T02:03:50.902 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-04-19T02:03:50.903 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.903 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:344 3.4s0 3:2277bf56:::smithi05930959-44:head [append 0~606208] snapc 8=[8,6,5] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:50.903 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.735396+0000", 2024-04-19T02:03:50.903 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 26.167928740000001, 2024-04-19T02:03:50.903 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.014124197999999999, 2024-04-19T02:03:50.903 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.903 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.903 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:50.903 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:50.903 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:50.903 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:50.903 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 344 2024-04-19T02:03:50.903 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.903 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.904 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.904 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.904 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.735396+0000", 2024-04-19T02:03:50.904 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.904 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.904 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.904 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.904 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.735396+0000", 2024-04-19T02:03:50.904 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.904 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.904 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.904 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.904 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.735399+0000", 2024-04-19T02:03:50.904 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.32e-06 2024-04-19T02:03:50.904 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.905 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.905 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.905 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.736033+0000", 2024-04-19T02:03:50.905 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000633689 2024-04-19T02:03:50.905 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.905 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.905 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.905 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.736074+0000", 2024-04-19T02:03:50.905 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1020999999999999e-05 2024-04-19T02:03:50.905 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.905 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.905 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.905 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.736094+0000", 2024-04-19T02:03:50.906 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0339e-05 2024-04-19T02:03:50.906 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.906 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.906 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.906 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.736202+0000", 2024-04-19T02:03:50.906 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000107899 2024-04-19T02:03:50.906 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.906 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.906 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "op must wait for map", 2024-04-19T02:03:50.906 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.736212+0000", 2024-04-19T02:03:50.906 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.7259999999999997e-06 2024-04-19T02:03:50.906 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.906 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.906 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.907 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.738290+0000", 2024-04-19T02:03:50.907 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002078725 2024-04-19T02:03:50.907 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.907 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.907 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:50.907 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.738359+0000", 2024-04-19T02:03:50.907 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8177999999999997e-05 2024-04-19T02:03:50.907 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.907 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.907 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:50.907 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.739412+0000", 2024-04-19T02:03:50.907 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0010534240000000001 2024-04-19T02:03:50.907 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.907 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.907 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:50.908 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.741318+0000", 2024-04-19T02:03:50.908 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001906151 2024-04-19T02:03:50.908 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.908 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.908 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:50.908 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749459+0000", 2024-04-19T02:03:50.908 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0081410830000000003 2024-04-19T02:03:50.908 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.908 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.908 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.908 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749520+0000", 2024-04-19T02:03:50.908 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0643000000000002e-05 2024-04-19T02:03:50.908 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.908 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.908 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.909 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.909 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.909 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:346 3.4s0 3:2277bf56:::smithi05930959-44:head [append 0~49152] snapc 8=[8,6,5] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:50.909 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.746194+0000", 2024-04-19T02:03:50.909 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 26.157130505000001, 2024-04-19T02:03:50.909 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.009380457, 2024-04-19T02:03:50.909 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.909 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.909 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:50.909 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:50.909 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:50.909 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:50.909 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 346 2024-04-19T02:03:50.910 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.910 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.910 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.910 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.910 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746194+0000", 2024-04-19T02:03:50.910 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.910 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.910 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.910 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.910 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746194+0000", 2024-04-19T02:03:50.910 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.910 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.910 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.911 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.911 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746195+0000", 2024-04-19T02:03:50.911 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.6199999999999997e-07 2024-04-19T02:03:50.911 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.911 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.911 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.911 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746238+0000", 2024-04-19T02:03:50.911 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.3105999999999997e-05 2024-04-19T02:03:50.911 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.911 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.911 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.911 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746247+0000", 2024-04-19T02:03:50.911 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.1540000000000008e-06 2024-04-19T02:03:50.911 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.911 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.912 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.912 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746257+0000", 2024-04-19T02:03:50.912 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0558e-05 2024-04-19T02:03:50.912 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.912 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.912 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.912 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.748250+0000", 2024-04-19T02:03:50.912 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0019923380000000002 2024-04-19T02:03:50.912 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.912 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.912 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:50.912 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.748349+0000", 2024-04-19T02:03:50.912 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9030999999999994e-05 2024-04-19T02:03:50.912 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.913 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.913 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:50.913 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.748987+0000", 2024-04-19T02:03:50.913 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00063871800000000003 2024-04-19T02:03:50.913 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.913 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.913 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:50.913 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750226+0000", 2024-04-19T02:03:50.913 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0012389250000000001 2024-04-19T02:03:50.913 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.913 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.913 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:50.913 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.755523+0000", 2024-04-19T02:03:50.913 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.005297024 2024-04-19T02:03:50.914 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.914 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.914 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.914 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.755574+0000", 2024-04-19T02:03:50.914 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.0940999999999999e-05 2024-04-19T02:03:50.914 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.914 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.914 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.914 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.914 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.914 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:347 3.4s0 3:2277bf56:::smithi05930959-44:head [setxattr _header (56) in=7b] snapc 8=[8,6,5] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:50.914 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.746274+0000", 2024-04-19T02:03:50.914 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 26.157050325, 2024-04-19T02:03:50.914 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010236558, 2024-04-19T02:03:50.915 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.915 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.915 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:50.915 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:50.915 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:50.915 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:50.915 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 347 2024-04-19T02:03:50.915 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.915 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.915 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.915 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.915 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746274+0000", 2024-04-19T02:03:50.915 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.915 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.916 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.916 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.916 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746274+0000", 2024-04-19T02:03:50.916 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.916 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.916 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.916 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.916 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746275+0000", 2024-04-19T02:03:50.916 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.3900000000000004e-07 2024-04-19T02:03:50.916 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.916 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.916 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.916 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746279+0000", 2024-04-19T02:03:50.917 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0679999999999998e-06 2024-04-19T02:03:50.917 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.917 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.917 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.917 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746299+0000", 2024-04-19T02:03:50.917 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9641e-05 2024-04-19T02:03:50.917 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.917 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.917 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.917 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746307+0000", 2024-04-19T02:03:50.917 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2570000000000007e-06 2024-04-19T02:03:50.918 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.918 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.918 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.918 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749555+0000", 2024-04-19T02:03:50.918 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0032478519999999999 2024-04-19T02:03:50.918 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.918 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.918 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:50.918 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749610+0000", 2024-04-19T02:03:50.918 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5155999999999998e-05 2024-04-19T02:03:50.918 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.918 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.918 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:50.918 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749969+0000", 2024-04-19T02:03:50.919 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00035909400000000001 2024-04-19T02:03:50.919 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.919 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.919 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:50.919 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750800+0000", 2024-04-19T02:03:50.919 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00083078000000000004 2024-04-19T02:03:50.919 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.919 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.919 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:50.919 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.756433+0000", 2024-04-19T02:03:50.919 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0056334009999999997 2024-04-19T02:03:50.919 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.919 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.919 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.920 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.756511+0000", 2024-04-19T02:03:50.920 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.7570000000000004e-05 2024-04-19T02:03:50.920 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.920 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.920 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.920 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.920 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.920 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:348 3.4s0 3:2277bf56:::smithi05930959-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:50.920 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.746345+0000", 2024-04-19T02:03:50.920 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 26.15697952, 2024-04-19T02:03:50.920 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012391231000000001, 2024-04-19T02:03:50.920 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.920 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.920 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:50.920 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:50.921 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:50.921 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:50.921 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 348 2024-04-19T02:03:50.921 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.921 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.921 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.921 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.921 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746345+0000", 2024-04-19T02:03:50.921 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.921 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.921 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.921 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.921 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746345+0000", 2024-04-19T02:03:50.921 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.921 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.922 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.922 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.922 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746346+0000", 2024-04-19T02:03:50.922 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.5899999999999996e-07 2024-04-19T02:03:50.922 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.922 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.922 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.922 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746349+0000", 2024-04-19T02:03:50.922 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2100000000000002e-06 2024-04-19T02:03:50.922 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.922 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.922 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.922 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746359+0000", 2024-04-19T02:03:50.922 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.8069999999999994e-06 2024-04-19T02:03:50.923 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.923 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.923 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.923 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746366+0000", 2024-04-19T02:03:50.923 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.1969999999999999e-06 2024-04-19T02:03:50.923 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.923 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.923 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.923 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750173+0000", 2024-04-19T02:03:50.923 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0038071400000000001 2024-04-19T02:03:50.923 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.923 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.923 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:50.923 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750223+0000", 2024-04-19T02:03:50.923 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9597e-05 2024-04-19T02:03:50.924 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.924 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.924 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.924 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.757010+0000", 2024-04-19T02:03:50.924 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.006787062 2024-04-19T02:03:50.924 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.924 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.924 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:50.924 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.757155+0000", 2024-04-19T02:03:50.924 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000145303 2024-04-19T02:03:50.924 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.924 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.924 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.924 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.758736+0000", 2024-04-19T02:03:50.924 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0015812560000000001 2024-04-19T02:03:50.925 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.925 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.925 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.925 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.925 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.925 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:374 3.4s0 3:2277bf56:::smithi05930959-44:head [append 0~0] snapc 9=[9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:50.925 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:26.758473+0000", 2024-04-19T02:03:50.925 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 24.144851273, 2024-04-19T02:03:50.925 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.032817486999999999, 2024-04-19T02:03:50.925 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.925 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.925 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:50.925 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:50.925 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:50.925 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:50.926 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 374 2024-04-19T02:03:50.926 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.926 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.926 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.926 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.926 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758473+0000", 2024-04-19T02:03:50.926 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.926 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.926 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.926 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.926 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758473+0000", 2024-04-19T02:03:50.926 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.926 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.926 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.927 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.927 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758475+0000", 2024-04-19T02:03:50.927 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.835e-06 2024-04-19T02:03:50.927 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.927 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.927 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.927 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758481+0000", 2024-04-19T02:03:50.927 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0669999999999997e-06 2024-04-19T02:03:50.927 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.927 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.927 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.927 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758497+0000", 2024-04-19T02:03:50.927 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6195e-05 2024-04-19T02:03:50.927 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.927 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.927 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.928 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758506+0000", 2024-04-19T02:03:50.928 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.1730000000000001e-06 2024-04-19T02:03:50.928 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.928 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.928 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.928 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758624+0000", 2024-04-19T02:03:50.928 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000117336 2024-04-19T02:03:50.928 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.928 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.928 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:50.928 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758812+0000", 2024-04-19T02:03:50.928 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000188684 2024-04-19T02:03:50.928 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.928 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.928 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:50.929 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.760058+0000", 2024-04-19T02:03:50.929 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001245812 2024-04-19T02:03:50.929 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.929 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.929 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:50.929 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.762381+0000", 2024-04-19T02:03:50.929 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0023226549999999999 2024-04-19T02:03:50.929 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.929 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.929 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:50.929 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.791180+0000", 2024-04-19T02:03:50.929 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.028798727 2024-04-19T02:03:50.929 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.929 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.930 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.930 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.791291+0000", 2024-04-19T02:03:50.930 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00011100299999999999 2024-04-19T02:03:50.930 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.930 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.930 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.930 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.930 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.930 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:375 3.4s0 3:2277bf56:::smithi05930959-44:head [rollback 8] snapc 9=[9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:50.930 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:26.758531+0000", 2024-04-19T02:03:50.930 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 24.144793400000001, 2024-04-19T02:03:50.930 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.033180481999999997, 2024-04-19T02:03:50.930 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.930 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.931 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:50.931 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:50.931 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:50.931 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:50.931 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 375 2024-04-19T02:03:50.931 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.931 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.931 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.931 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.931 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758531+0000", 2024-04-19T02:03:50.931 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.931 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.931 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.931 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.932 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758531+0000", 2024-04-19T02:03:50.932 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.932 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.932 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.932 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.932 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758532+0000", 2024-04-19T02:03:50.932 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.5700000000000002e-07 2024-04-19T02:03:50.932 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.932 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.932 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.932 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758535+0000", 2024-04-19T02:03:50.932 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4520000000000002e-06 2024-04-19T02:03:50.932 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.932 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.933 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.933 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758546+0000", 2024-04-19T02:03:50.933 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1195e-05 2024-04-19T02:03:50.933 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.933 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.933 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.933 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758554+0000", 2024-04-19T02:03:50.933 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.2579999999999998e-06 2024-04-19T02:03:50.933 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.933 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.933 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.933 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.760997+0000", 2024-04-19T02:03:50.933 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0024429930000000001 2024-04-19T02:03:50.933 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.934 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.934 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:50.934 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.761153+0000", 2024-04-19T02:03:50.934 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00015617100000000001 2024-04-19T02:03:50.934 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.934 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.934 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:50.934 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.761823+0000", 2024-04-19T02:03:50.934 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00067058000000000005 2024-04-19T02:03:50.934 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.934 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.934 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:50.934 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.765008+0000", 2024-04-19T02:03:50.934 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0031845300000000001 2024-04-19T02:03:50.934 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.935 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.935 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:50.935 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.791630+0000", 2024-04-19T02:03:50.935 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.026622456999999999 2024-04-19T02:03:50.935 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.935 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.935 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.935 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.791712+0000", 2024-04-19T02:03:50.935 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.1188999999999998e-05 2024-04-19T02:03:50.935 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.935 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.935 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.935 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.935 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.935 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:376 3.4s0 3:2277bf56:::smithi05930959-44:head [append 0~0] snapc 9=[9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:50.936 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:26.758565+0000", 2024-04-19T02:03:50.936 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 24.144759642, 2024-04-19T02:03:50.936 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.033486091000000003, 2024-04-19T02:03:50.936 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.936 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.936 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:50.936 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:50.936 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:50.936 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:50.936 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 376 2024-04-19T02:03:50.936 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.936 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.936 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.936 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.936 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758565+0000", 2024-04-19T02:03:50.937 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.937 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.937 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.937 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.937 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758565+0000", 2024-04-19T02:03:50.937 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.937 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.937 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.937 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.937 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758565+0000", 2024-04-19T02:03:50.937 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9900000000000001e-07 2024-04-19T02:03:50.937 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.937 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.937 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.937 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758568+0000", 2024-04-19T02:03:50.938 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0010000000000002e-06 2024-04-19T02:03:50.938 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.938 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.938 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.938 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758576+0000", 2024-04-19T02:03:50.938 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.9719999999999992e-06 2024-04-19T02:03:50.938 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.938 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.938 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.938 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758583+0000", 2024-04-19T02:03:50.938 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.2489999999999998e-06 2024-04-19T02:03:50.938 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.938 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.938 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.939 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.763053+0000", 2024-04-19T02:03:50.939 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0044703809999999998 2024-04-19T02:03:50.939 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.939 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.939 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:50.939 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.763266+0000", 2024-04-19T02:03:50.939 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00021301299999999999 2024-04-19T02:03:50.939 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.939 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.939 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:50.939 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.764685+0000", 2024-04-19T02:03:50.939 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0014195379999999999 2024-04-19T02:03:50.939 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.939 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.939 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:50.940 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.765894+0000", 2024-04-19T02:03:50.940 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0012088859999999999 2024-04-19T02:03:50.940 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.940 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.940 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:50.940 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.791938+0000", 2024-04-19T02:03:50.940 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.026043251999999999 2024-04-19T02:03:50.940 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.940 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.940 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.940 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.792051+0000", 2024-04-19T02:03:50.940 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0001133 2024-04-19T02:03:50.940 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.940 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.940 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.941 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.941 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.941 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.as2 46/26 ECSubWrite(tid=102, reqid=osd.11.0:102, at_version=46'47, trim_to=0'0, roll_forward_to=45'45))", 2024-04-19T02:03:50.941 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.061978+0000", 2024-04-19T02:03:50.941 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 20.841346516000002, 2024-04-19T02:03:50.941 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0086516360000000007, 2024-04-19T02:03:50.941 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.941 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.941 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:50.941 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.941 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.941 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.941 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.061978+0000", 2024-04-19T02:03:50.941 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.941 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.941 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.942 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.942 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.061978+0000", 2024-04-19T02:03:50.942 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.942 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.942 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.942 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.942 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.061979+0000", 2024-04-19T02:03:50.942 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.8000000000000004e-07 2024-04-19T02:03:50.942 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.942 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.942 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.942 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.062004+0000", 2024-04-19T02:03:50.942 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.531e-05 2024-04-19T02:03:50.942 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.943 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.943 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.943 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.062022+0000", 2024-04-19T02:03:50.943 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8000999999999999e-05 2024-04-19T02:03:50.943 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.943 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.943 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.943 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.062036+0000", 2024-04-19T02:03:50.943 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3499999999999999e-05 2024-04-19T02:03:50.943 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.943 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.944 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.944 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.062138+0000", 2024-04-19T02:03:50.944 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010238099999999999 2024-04-19T02:03:50.944 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.944 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.944 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:50.944 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.062149+0000", 2024-04-19T02:03:50.944 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0706e-05 2024-04-19T02:03:50.944 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.944 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.944 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:50.944 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.070585+0000", 2024-04-19T02:03:50.945 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0084365910000000002 2024-04-19T02:03:50.945 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.945 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.945 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.945 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.070630+0000", 2024-04-19T02:03:50.945 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4267000000000002e-05 2024-04-19T02:03:50.945 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.945 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.945 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.945 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.945 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.945 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.8s1 46/26 ECSubWrite(tid=81, reqid=osd.4.0:81, at_version=46'42, trim_to=0'0, roll_forward_to=45'40))", 2024-04-19T02:03:50.946 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.062231+0000", 2024-04-19T02:03:50.946 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 20.841093522000001, 2024-04-19T02:03:50.946 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0074011800000000003, 2024-04-19T02:03:50.946 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.946 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.946 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:50.946 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.946 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.946 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.946 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.062231+0000", 2024-04-19T02:03:50.947 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.947 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.947 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.947 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.947 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.062231+0000", 2024-04-19T02:03:50.947 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.947 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.947 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.947 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.947 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.062231+0000", 2024-04-19T02:03:50.947 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.96e-07 2024-04-19T02:03:50.947 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.948 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.948 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.948 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.062253+0000", 2024-04-19T02:03:50.948 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2266e-05 2024-04-19T02:03:50.948 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.948 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.948 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.948 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.062266+0000", 2024-04-19T02:03:50.948 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2784e-05 2024-04-19T02:03:50.948 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.948 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.948 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.949 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.062275+0000", 2024-04-19T02:03:50.949 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7069999999999998e-06 2024-04-19T02:03:50.949 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.949 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.949 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.949 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.062378+0000", 2024-04-19T02:03:50.949 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000103473 2024-04-19T02:03:50.949 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.949 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.949 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:50.949 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.062388+0000", 2024-04-19T02:03:50.950 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9170000000000001e-06 2024-04-19T02:03:50.950 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.950 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.950 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:50.950 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.069586+0000", 2024-04-19T02:03:50.950 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0071980660000000004 2024-04-19T02:03:50.950 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.950 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.950 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.950 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.069632+0000", 2024-04-19T02:03:50.950 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5670999999999997e-05 2024-04-19T02:03:50.950 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.951 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.951 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.951 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.951 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.951 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.as2 46/26 ECSubWrite(tid=103, reqid=osd.11.0:103, at_version=46'49, trim_to=0'0, roll_forward_to=45'45))", 2024-04-19T02:03:50.951 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.069253+0000", 2024-04-19T02:03:50.951 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 20.834071915999999, 2024-04-19T02:03:50.951 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0096013669999999995, 2024-04-19T02:03:50.951 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.951 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.951 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:50.952 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.952 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.952 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.952 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.069253+0000", 2024-04-19T02:03:50.952 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.952 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.952 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.952 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.952 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.069253+0000", 2024-04-19T02:03:50.952 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.952 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.952 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.952 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.953 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.069253+0000", 2024-04-19T02:03:50.953 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.0100000000000004e-07 2024-04-19T02:03:50.953 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.953 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.953 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.953 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.069282+0000", 2024-04-19T02:03:50.953 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8617999999999999e-05 2024-04-19T02:03:50.953 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.953 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.953 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.953 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.069293+0000", 2024-04-19T02:03:50.953 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1276e-05 2024-04-19T02:03:50.953 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.954 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.954 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.954 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.069302+0000", 2024-04-19T02:03:50.954 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.8910000000000001e-06 2024-04-19T02:03:50.954 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.954 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.954 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.954 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.069822+0000", 2024-04-19T02:03:50.954 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00051964899999999996 2024-04-19T02:03:50.954 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.954 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.955 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:50.955 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.069835+0000", 2024-04-19T02:03:50.955 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.358e-05 2024-04-19T02:03:50.955 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.955 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.955 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:50.955 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.078808+0000", 2024-04-19T02:03:50.955 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0089730929999999997 2024-04-19T02:03:50.955 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.955 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.955 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.955 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.078854+0000", 2024-04-19T02:03:50.955 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5559000000000002e-05 2024-04-19T02:03:50.955 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.955 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.956 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.956 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.956 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.956 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.7s7 46/26 ECSubWrite(tid=158, reqid=osd.3.0:158, at_version=46'14, trim_to=0'0, roll_forward_to=45'12))", 2024-04-19T02:03:50.956 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.071853+0000", 2024-04-19T02:03:50.956 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 20.831471008000001, 2024-04-19T02:03:50.956 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.009672043, 2024-04-19T02:03:50.956 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.956 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.956 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:50.956 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.956 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.956 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.956 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.071853+0000", 2024-04-19T02:03:50.957 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.957 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.957 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.957 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.957 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.071853+0000", 2024-04-19T02:03:50.957 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.957 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.957 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.957 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.957 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.071854+0000", 2024-04-19T02:03:50.957 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1600000000000002e-07 2024-04-19T02:03:50.957 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.957 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.957 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.958 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.071880+0000", 2024-04-19T02:03:50.958 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6648e-05 2024-04-19T02:03:50.958 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.958 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.958 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.958 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.071895+0000", 2024-04-19T02:03:50.958 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4491999999999999e-05 2024-04-19T02:03:50.958 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.958 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.958 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.958 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.071915+0000", 2024-04-19T02:03:50.958 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9762000000000002e-05 2024-04-19T02:03:50.958 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.959 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.959 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.959 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.071997+0000", 2024-04-19T02:03:50.959 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2693000000000003e-05 2024-04-19T02:03:50.959 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.959 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.959 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:50.959 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.072009+0000", 2024-04-19T02:03:50.959 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2004000000000001e-05 2024-04-19T02:03:50.959 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.959 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.959 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:50.959 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.081483+0000", 2024-04-19T02:03:50.959 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0094739990000000003 2024-04-19T02:03:50.960 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.960 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.960 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.960 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.081525+0000", 2024-04-19T02:03:50.960 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2029e-05 2024-04-19T02:03:50.960 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.960 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.960 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.960 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.960 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.960 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:483 3.4s0 3:2859204c:::smithi05930959-39:head [setxattr _header (56) in=7b] snapc a=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:03:50.960 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:31.214133+0000", 2024-04-19T02:03:50.960 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 19.689191893, 2024-04-19T02:03:50.960 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0073982509999999998, 2024-04-19T02:03:50.961 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.961 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.961 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:50.961 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:50.961 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:50.961 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:50.961 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 483 2024-04-19T02:03:50.961 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.961 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.961 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.961 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.961 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.214133+0000", 2024-04-19T02:03:50.961 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.961 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.961 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.962 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.962 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.214133+0000", 2024-04-19T02:03:50.962 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.962 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.962 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.962 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.962 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.214133+0000", 2024-04-19T02:03:50.962 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.0699999999999996e-07 2024-04-19T02:03:50.962 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.962 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.962 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.962 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.214139+0000", 2024-04-19T02:03:50.962 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.4229999999999996e-06 2024-04-19T02:03:50.962 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.963 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.963 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.963 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.214148+0000", 2024-04-19T02:03:50.963 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.5540000000000003e-06 2024-04-19T02:03:50.963 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.963 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.963 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.963 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.214154+0000", 2024-04-19T02:03:50.963 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.2439999999999998e-06 2024-04-19T02:03:50.963 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.963 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.963 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.963 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.216962+0000", 2024-04-19T02:03:50.963 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002807866 2024-04-19T02:03:50.963 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.964 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.964 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:50.964 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.217064+0000", 2024-04-19T02:03:50.964 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000101216 2024-04-19T02:03:50.964 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.964 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.964 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:50.964 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.217542+0000", 2024-04-19T02:03:50.964 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00047826900000000002 2024-04-19T02:03:50.964 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.964 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.964 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:50.964 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.219838+0000", 2024-04-19T02:03:50.964 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002296574 2024-04-19T02:03:50.965 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.965 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.965 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:50.965 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.221438+0000", 2024-04-19T02:03:50.965 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0015996890000000001 2024-04-19T02:03:50.965 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.965 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.965 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.965 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.221531+0000", 2024-04-19T02:03:50.965 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.2708999999999996e-05 2024-04-19T02:03:50.965 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.966 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.966 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.966 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.966 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.966 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:484 3.4s0 3:2859204c:::smithi05930959-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:03:50.966 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:31.214166+0000", 2024-04-19T02:03:50.966 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 19.689158489, 2024-04-19T02:03:50.966 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010260811999999999, 2024-04-19T02:03:50.966 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.966 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.966 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:50.966 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:50.966 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:50.967 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:50.967 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 484 2024-04-19T02:03:50.967 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.967 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.967 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.967 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.967 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.214166+0000", 2024-04-19T02:03:50.967 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.967 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.967 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.967 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.967 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.214166+0000", 2024-04-19T02:03:50.967 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.968 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.968 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.968 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.968 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.214167+0000", 2024-04-19T02:03:50.968 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7000000000000005e-07 2024-04-19T02:03:50.968 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.968 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.968 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.968 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.214169+0000", 2024-04-19T02:03:50.968 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6240000000000002e-06 2024-04-19T02:03:50.968 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.968 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.969 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.969 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.214178+0000", 2024-04-19T02:03:50.969 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.721e-06 2024-04-19T02:03:50.969 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.969 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.969 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.969 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.214187+0000", 2024-04-19T02:03:50.969 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.5710000000000004e-06 2024-04-19T02:03:50.969 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.969 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.970 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.970 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.217834+0000", 2024-04-19T02:03:50.970 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.003646846 2024-04-19T02:03:50.970 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.970 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.970 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:50.970 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.217931+0000", 2024-04-19T02:03:50.970 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.6584999999999998e-05 2024-04-19T02:03:50.970 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.970 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.970 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.971 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.222283+0000", 2024-04-19T02:03:50.971 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0043518469999999998 2024-04-19T02:03:50.971 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.971 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.971 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:50.971 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.222384+0000", 2024-04-19T02:03:50.971 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000101426 2024-04-19T02:03:50.971 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.971 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.971 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.971 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.224427+0000", 2024-04-19T02:03:50.971 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002042622 2024-04-19T02:03:50.971 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.972 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.972 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.972 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.972 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.972 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.8s1 57/26 ECSubWrite(tid=120, reqid=osd.4.0:120, at_version=57'63, trim_to=0'0, roll_forward_to=56'61))", 2024-04-19T02:03:50.972 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:41.151500+0000", 2024-04-19T02:03:50.972 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 9.751824633, 2024-04-19T02:03:50.972 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.007592934, 2024-04-19T02:03:50.972 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.972 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.972 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:03:50.972 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.972 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.973 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.973 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.151500+0000", 2024-04-19T02:03:50.973 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.973 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.973 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.973 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.973 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.151500+0000", 2024-04-19T02:03:50.973 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.973 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.973 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.973 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.973 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.151500+0000", 2024-04-19T02:03:50.973 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.03e-07 2024-04-19T02:03:50.974 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.974 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.974 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.974 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.151537+0000", 2024-04-19T02:03:50.974 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.7073999999999998e-05 2024-04-19T02:03:50.974 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.974 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.974 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.974 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.151552+0000", 2024-04-19T02:03:50.974 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5129e-05 2024-04-19T02:03:50.974 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.975 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.975 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.975 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.151564+0000", 2024-04-19T02:03:50.975 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1894e-05 2024-04-19T02:03:50.975 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.975 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.975 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.975 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.151654+0000", 2024-04-19T02:03:50.975 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.9244000000000001e-05 2024-04-19T02:03:50.975 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.975 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.975 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:50.975 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.151670+0000", 2024-04-19T02:03:50.976 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6341000000000002e-05 2024-04-19T02:03:50.976 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.976 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.976 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:50.976 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.159033+0000", 2024-04-19T02:03:50.976 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0073635310000000004 2024-04-19T02:03:50.976 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.976 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.976 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.976 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.159093+0000", 2024-04-19T02:03:50.977 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.9317999999999997e-05 2024-04-19T02:03:50.977 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.977 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.977 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.977 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.977 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.977 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:634 3.4s0 3:2277bf56:::smithi05930959-44:head [delete] snapc 13=[13,12,10,c] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e61)", 2024-04-19T02:03:50.977 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:45.174844+0000", 2024-04-19T02:03:50.977 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 5.7284800059999998, 2024-04-19T02:03:50.977 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0099451439999999995, 2024-04-19T02:03:50.978 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.978 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.978 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:50.978 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:50.978 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:50.978 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:50.978 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 634 2024-04-19T02:03:50.978 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.978 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.978 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.978 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.978 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:45.174844+0000", 2024-04-19T02:03:50.978 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.979 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.979 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.979 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.979 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:45.174844+0000", 2024-04-19T02:03:50.979 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.979 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.979 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.979 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.979 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:45.174847+0000", 2024-04-19T02:03:50.979 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7989999999999998e-06 2024-04-19T02:03:50.979 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.979 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.980 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.980 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:45.174859+0000", 2024-04-19T02:03:50.980 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1717e-05 2024-04-19T02:03:50.980 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.980 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.980 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.980 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:45.174883+0000", 2024-04-19T02:03:50.980 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3907999999999999e-05 2024-04-19T02:03:50.980 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.980 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.980 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.980 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:45.174895+0000", 2024-04-19T02:03:50.980 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1878999999999999e-05 2024-04-19T02:03:50.980 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.981 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.981 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.981 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:45.174988+0000", 2024-04-19T02:03:50.981 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3517000000000006e-05 2024-04-19T02:03:50.981 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.981 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.981 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "op must wait for map", 2024-04-19T02:03:50.981 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:45.174998+0000", 2024-04-19T02:03:50.981 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3929999999999997e-06 2024-04-19T02:03:50.981 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.981 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.981 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.981 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:45.177575+0000", 2024-04-19T02:03:50.981 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0025775860000000002 2024-04-19T02:03:50.982 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.982 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.982 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:50.982 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:45.177661+0000", 2024-04-19T02:03:50.982 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.6163999999999997e-05 2024-04-19T02:03:50.982 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.982 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.982 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:50.982 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:45.178434+0000", 2024-04-19T02:03:50.982 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00077293999999999996 2024-04-19T02:03:50.982 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.982 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.983 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:50.983 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:45.181696+0000", 2024-04-19T02:03:50.983 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0032614929999999999 2024-04-19T02:03:50.983 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.983 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.983 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:50.983 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:45.184701+0000", 2024-04-19T02:03:50.983 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0030054209999999999 2024-04-19T02:03:50.983 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.983 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.983 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.983 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:45.184790+0000", 2024-04-19T02:03:50.983 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.8326999999999996e-05 2024-04-19T02:03:50.983 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.984 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.984 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.984 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.984 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.984 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:728 3.4s0 3:2277bf56:::smithi05930959-44:head [delete] snapc 16=[16,15,12,10,c] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e66)", 2024-04-19T02:03:50.984 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:50.059596+0000", 2024-04-19T02:03:50.984 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.84372871299999996, 2024-04-19T02:03:50.984 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.015090878, 2024-04-19T02:03:50.984 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.984 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.984 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:03:50.984 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:50.984 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:50.985 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:50.985 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 728 2024-04-19T02:03:50.985 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.985 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.985 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.985 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.985 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.059596+0000", 2024-04-19T02:03:50.985 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.985 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.985 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.985 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.985 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.059596+0000", 2024-04-19T02:03:50.985 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.986 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.986 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.986 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.986 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.059598+0000", 2024-04-19T02:03:50.986 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5160000000000001e-06 2024-04-19T02:03:50.986 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.986 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.986 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.986 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.059609+0000", 2024-04-19T02:03:50.986 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1051e-05 2024-04-19T02:03:50.986 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.987 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.987 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.987 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.059635+0000", 2024-04-19T02:03:50.987 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6185000000000001e-05 2024-04-19T02:03:50.987 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.987 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.987 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.987 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.059648+0000", 2024-04-19T02:03:50.987 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2439999999999999e-05 2024-04-19T02:03:50.987 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.987 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.987 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.987 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.059734+0000", 2024-04-19T02:03:50.987 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.6207999999999999e-05 2024-04-19T02:03:50.988 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.988 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.988 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "op must wait for map", 2024-04-19T02:03:50.988 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.059743+0000", 2024-04-19T02:03:50.988 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.2520000000000005e-06 2024-04-19T02:03:50.988 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.988 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.988 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.988 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.061974+0000", 2024-04-19T02:03:50.988 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0022307889999999999 2024-04-19T02:03:50.988 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.988 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.989 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:50.989 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.062111+0000", 2024-04-19T02:03:50.989 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013688799999999999 2024-04-19T02:03:50.989 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.989 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.989 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.989 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.074687+0000", 2024-04-19T02:03:50.989 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012575549 2024-04-19T02:03:50.989 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.989 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.989 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.989 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.989 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.990 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:735 3.4s0 3:24202781:::smithi05930959-26:head [append 0~0] snapc 16=[16,15,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-19T02:03:50.990 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:50.075102+0000", 2024-04-19T02:03:50.990 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.82822246700000002, 2024-04-19T02:03:50.990 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010483856, 2024-04-19T02:03:50.990 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.990 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.990 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:50.990 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:50.990 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:50.990 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:50.990 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 735 2024-04-19T02:03:50.990 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.991 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.991 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.991 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.991 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.075102+0000", 2024-04-19T02:03:50.991 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.991 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.991 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.991 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.991 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.075102+0000", 2024-04-19T02:03:50.991 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.991 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.991 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.992 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.992 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.075104+0000", 2024-04-19T02:03:50.992 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.858e-06 2024-04-19T02:03:50.992 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.992 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.992 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.992 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.075111+0000", 2024-04-19T02:03:50.992 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.985e-06 2024-04-19T02:03:50.992 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.992 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.992 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.992 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.075128+0000", 2024-04-19T02:03:50.992 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6824999999999999e-05 2024-04-19T02:03:50.993 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.993 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.993 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.993 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.075136+0000", 2024-04-19T02:03:50.993 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7879999999999995e-06 2024-04-19T02:03:50.993 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.993 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.993 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.993 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.075207+0000", 2024-04-19T02:03:50.993 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.0865000000000007e-05 2024-04-19T02:03:50.993 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.993 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.993 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:50.994 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.075341+0000", 2024-04-19T02:03:50.994 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000133535 2024-04-19T02:03:50.994 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.994 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.994 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:50.994 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.076246+0000", 2024-04-19T02:03:50.994 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00090560300000000005 2024-04-19T02:03:50.994 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.994 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.994 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:50.994 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.079011+0000", 2024-04-19T02:03:50.994 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0027648999999999998 2024-04-19T02:03:50.995 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.995 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.995 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:50.995 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.085466+0000", 2024-04-19T02:03:50.995 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0064548959999999999 2024-04-19T02:03:50.995 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.995 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.995 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:50.995 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.085586+0000", 2024-04-19T02:03:50.995 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000119601 2024-04-19T02:03:50.995 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.995 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:50.995 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:50.996 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.996 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.996 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:736 3.4s0 3:24202781:::smithi05930959-26:head [rollback c] snapc 16=[16,15,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-19T02:03:50.996 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:50.075157+0000", 2024-04-19T02:03:50.996 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.82816699299999996, 2024-04-19T02:03:50.996 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.01141724, 2024-04-19T02:03:50.996 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:50.996 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:50.996 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:50.996 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:50.996 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:50.996 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:50.996 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 736 2024-04-19T02:03:50.997 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.997 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:50.997 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.997 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:50.997 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.075157+0000", 2024-04-19T02:03:50.997 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.997 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.997 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.997 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:50.997 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.075157+0000", 2024-04-19T02:03:50.997 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:50.997 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.997 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.998 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:50.998 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.075158+0000", 2024-04-19T02:03:50.998 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.8400000000000004e-07 2024-04-19T02:03:50.998 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.998 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.998 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:50.998 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.075161+0000", 2024-04-19T02:03:50.998 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.174e-06 2024-04-19T02:03:50.998 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.998 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.998 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:50.998 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.075180+0000", 2024-04-19T02:03:50.998 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9228999999999998e-05 2024-04-19T02:03:50.998 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.999 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.999 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:50.999 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.075188+0000", 2024-04-19T02:03:50.999 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6539999999999999e-06 2024-04-19T02:03:50.999 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.999 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.999 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:50.999 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.077039+0000", 2024-04-19T02:03:50.999 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001850499 2024-04-19T02:03:50.999 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:50.999 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:50.999 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:50.999 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.077144+0000", 2024-04-19T02:03:51.000 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000105923 2024-04-19T02:03:51.000 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:51.000 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:51.000 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:51.000 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.077764+0000", 2024-04-19T02:03:51.000 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00061906400000000001 2024-04-19T02:03:51.000 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:51.000 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:51.000 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:51.000 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.079013+0000", 2024-04-19T02:03:51.000 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001249893 2024-04-19T02:03:51.000 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:51.001 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:51.001 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:51.001 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.086479+0000", 2024-04-19T02:03:51.001 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.007465541 2024-04-19T02:03:51.001 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:51.001 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:51.001 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:51.001 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.086575+0000", 2024-04-19T02:03:51.001 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.5679000000000003e-05 2024-04-19T02:03:51.001 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:51.001 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:51.001 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:51.001 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:51.002 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:51.002 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:737 3.4s0 3:24202781:::smithi05930959-26:head [append 0~0] snapc 16=[16,15,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-19T02:03:51.002 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:50.075201+0000", 2024-04-19T02:03:51.002 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.82812388999999997, 2024-04-19T02:03:51.002 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.014835278, 2024-04-19T02:03:51.002 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:51.002 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:51.002 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:51.002 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:51.002 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:51.002 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:51.002 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 737 2024-04-19T02:03:51.002 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:51.003 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:51.003 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:51.003 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:51.003 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.075201+0000", 2024-04-19T02:03:51.003 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:51.003 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:51.003 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:51.003 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:51.003 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.075201+0000", 2024-04-19T02:03:51.003 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:51.003 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:51.003 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:51.003 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:51.003 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.075201+0000", 2024-04-19T02:03:51.004 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.4300000000000003e-07 2024-04-19T02:03:51.004 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:51.004 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:51.004 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:51.004 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.075206+0000", 2024-04-19T02:03:51.004 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4669999999999998e-06 2024-04-19T02:03:51.004 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:51.004 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:51.004 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:51.004 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.075214+0000", 2024-04-19T02:03:51.004 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2949999999999994e-06 2024-04-19T02:03:51.004 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:51.004 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:51.004 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:51.005 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.075220+0000", 2024-04-19T02:03:51.005 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.4169999999999997e-06 2024-04-19T02:03:51.005 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:51.005 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:51.005 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:51.005 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.078117+0000", 2024-04-19T02:03:51.005 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0028966859999999999 2024-04-19T02:03:51.005 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:51.005 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:51.005 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:51.005 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.078218+0000", 2024-04-19T02:03:51.005 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010105799999999999 2024-04-19T02:03:51.005 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:51.005 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:51.006 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:51.006 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.078775+0000", 2024-04-19T02:03:51.006 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00055719600000000004 2024-04-19T02:03:51.006 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:51.006 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:51.006 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:51.006 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.079292+0000", 2024-04-19T02:03:51.006 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00051654700000000001 2024-04-19T02:03:51.006 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:51.006 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:51.006 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:51.006 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.089922+0000", 2024-04-19T02:03:51.006 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010630173999999999 2024-04-19T02:03:51.007 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:51.007 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:51.007 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:51.007 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.090036+0000", 2024-04-19T02:03:51.007 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00011389500000000001 2024-04-19T02:03:51.007 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:51.007 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:51.007 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:51.007 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:51.007 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:51.007 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:756 3.4s0 3:24169e7e:::smithi05930959-31:head [copy-from ver 2] snapc 16=[16,15,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-19T02:03:51.007 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:50.389774+0000", 2024-04-19T02:03:51.007 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 0.51355059400000003, 2024-04-19T02:03:51.007 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0130916, 2024-04-19T02:03:51.008 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:03:51.008 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:03:51.008 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:51.008 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:03:51.008 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:03:51.008 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:51.008 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 756 2024-04-19T02:03:51.008 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:51.008 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:03:51.008 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:51.008 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:03:51.008 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.389774+0000", 2024-04-19T02:03:51.008 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:51.009 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:51.009 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:51.009 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:03:51.009 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.389774+0000", 2024-04-19T02:03:51.009 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:03:51.009 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:51.009 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:51.009 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:03:51.009 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.389775+0000", 2024-04-19T02:03:51.009 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.604e-06 2024-04-19T02:03:51.009 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:51.009 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:51.009 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:03:51.009 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.389783+0000", 2024-04-19T02:03:51.010 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.1029999999999996e-06 2024-04-19T02:03:51.010 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:51.010 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:51.010 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:03:51.010 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.389796+0000", 2024-04-19T02:03:51.010 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3234999999999999e-05 2024-04-19T02:03:51.010 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:51.010 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:51.010 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:03:51.010 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.389803+0000", 2024-04-19T02:03:51.010 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9619999999999996e-06 2024-04-19T02:03:51.010 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:51.010 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:51.011 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:03:51.011 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.389876+0000", 2024-04-19T02:03:51.011 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.2898000000000006e-05 2024-04-19T02:03:51.011 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:51.011 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:51.011 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:03:51.011 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.390002+0000", 2024-04-19T02:03:51.011 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000126029 2024-04-19T02:03:51.011 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:51.011 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:51.011 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:03:51.011 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.395353+0000", 2024-04-19T02:03:51.011 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00535156 2024-04-19T02:03:51.012 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:51.012 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:51.012 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:03:51.012 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.396963+0000", 2024-04-19T02:03:51.012 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001609999 2024-04-19T02:03:51.012 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:51.012 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:51.012 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:03:51.012 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.402751+0000", 2024-04-19T02:03:51.012 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0057873000000000004 2024-04-19T02:03:51.012 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:03:51.012 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:03:51.012 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:03:51.012 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.402865+0000", 2024-04-19T02:03:51.013 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00011491 2024-04-19T02:03:51.013 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:51.013 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:51.013 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:51.013 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:03:51.013 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:03:51.013 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:51.015 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-19T02:03:51.015 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:51.017+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:51.058 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is unset 2024-04-19T02:03:51.066 INFO:tasks.rados.rados.0.smithi059.stdout:287: done (4 left) 2024-04-19T02:03:51.066 INFO:tasks.rados.rados.0.smithi059.stdout:288: done (3 left) 2024-04-19T02:03:51.066 INFO:tasks.rados.rados.0.smithi059.stdout:289: done (2 left) 2024-04-19T02:03:51.067 INFO:tasks.rados.rados.0.smithi059.stdout:290: done (1 left) 2024-04-19T02:03:51.067 INFO:tasks.rados.rados.0.smithi059.stdout:291: done (0 left) 2024-04-19T02:03:51.067 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 29 current snap is 22 2024-04-19T02:03:51.067 INFO:tasks.rados.rados.0.smithi059.stdout:292: seq_num 138 ranges {1687552=770048,2457600=540672,2998272=753664,3751936=262144} 2024-04-19T02:03:51.085 INFO:tasks.rados.rados.0.smithi059.stdout:292: writing smithi05930959-29 from 1687552 to 2457600 tid 1 2024-04-19T02:03:51.089 INFO:tasks.rados.rados.0.smithi059.stdout:292: writing smithi05930959-29 from 2457600 to 2998272 tid 2 2024-04-19T02:03:51.094 INFO:tasks.rados.rados.0.smithi059.stdout:292: writing smithi05930959-29 from 2998272 to 3751936 tid 3 2024-04-19T02:03:51.097 INFO:tasks.rados.rados.0.smithi059.stdout:292: writing smithi05930959-29 from 3751936 to 4014080 tid 4 2024-04-19T02:03:51.098 INFO:tasks.rados.rados.0.smithi059.stdout:293: read oid 4 snap 19 2024-04-19T02:03:51.098 INFO:tasks.rados.rados.0.smithi059.stdout:293: expect (ObjNum 126 snap 19 seq_num 126) 2024-04-19T02:03:51.098 INFO:tasks.rados.rados.0.smithi059.stdout:294: copy_from oid 20 from oid 11 current snap is 22 2024-04-19T02:03:51.098 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 41 current snap is 22 2024-04-19T02:03:51.098 INFO:tasks.rados.rados.0.smithi059.stdout:295: seq_num 139 ranges {5062656=786432,5849088=688128,6537216=475136,7012352=393216} 2024-04-19T02:03:51.116 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:51.117+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:51.122 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T02:03:51.141 INFO:tasks.rados.rados.0.smithi059.stdout:295: writing smithi05930959-41 from 5062656 to 5849088 tid 1 2024-04-19T02:03:51.145 INFO:tasks.rados.rados.0.smithi059.stdout:295: writing smithi05930959-41 from 5849088 to 6537216 tid 2 2024-04-19T02:03:51.148 INFO:tasks.rados.rados.0.smithi059.stdout:295: writing smithi05930959-41 from 6537216 to 7012352 tid 3 2024-04-19T02:03:51.151 INFO:tasks.rados.rados.0.smithi059.stdout:295: writing smithi05930959-41 from 7012352 to 7405568 tid 4 2024-04-19T02:03:51.151 INFO:tasks.rados.rados.0.smithi059.stdout:296: snap_create 2024-04-19T02:03:51.152 INFO:tasks.rados.rados.0.smithi059.stdout:294: finishing copy_from to smithi05930959-20 2024-04-19T02:03:51.152 INFO:tasks.rados.rados.0.smithi059.stdout:294: got expected ENOENT (src dne) 2024-04-19T02:03:51.152 INFO:tasks.rados.rados.0.smithi059.stdout:294: finishing copy_from racing read to smithi05930959-20 2024-04-19T02:03:51.152 INFO:tasks.rados.rados.0.smithi059.stdout:292: finishing write tid 1 to smithi05930959-29 2024-04-19T02:03:51.152 INFO:tasks.rados.rados.0.smithi059.stdout:292: finishing write tid 2 to smithi05930959-29 2024-04-19T02:03:51.152 INFO:tasks.rados.rados.0.smithi059.stdout:292: finishing write tid 3 to smithi05930959-29 2024-04-19T02:03:51.152 INFO:tasks.rados.rados.0.smithi059.stdout:292: finishing write tid 4 to smithi05930959-29 2024-04-19T02:03:51.152 INFO:tasks.rados.rados.0.smithi059.stdout:292: finishing write tid 5 to smithi05930959-29 2024-04-19T02:03:51.152 INFO:tasks.rados.rados.0.smithi059.stdout:292: finishing write tid 6 to smithi05930959-29 2024-04-19T02:03:51.152 INFO:tasks.rados.rados.0.smithi059.stdout:292: oid 29 updating version 0 to 45 2024-04-19T02:03:51.152 INFO:tasks.rados.rados.0.smithi059.stdout:292: oid 29 version 45 is already newer than 44 2024-04-19T02:03:51.152 INFO:tasks.rados.rados.0.smithi059.stdout:292: oid 29 version 45 is already newer than 43 2024-04-19T02:03:51.153 INFO:tasks.rados.rados.0.smithi059.stdout:292: oid 29 version 45 is already newer than 41 2024-04-19T02:03:51.153 INFO:tasks.rados.rados.0.smithi059.stdout:292: oid 29 version 45 is already newer than 42 2024-04-19T02:03:51.153 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 29 v 45 (ObjNum 138 snap 22 seq_num 138) dirty exists 2024-04-19T02:03:51.153 INFO:tasks.rados.rados.0.smithi059.stdout:292: left oid 29 (ObjNum 138 snap 22 seq_num 138) 2024-04-19T02:03:51.153 INFO:tasks.rados.rados.0.smithi059.stdout:293: expect (ObjNum 126 snap 19 seq_num 126) 2024-04-19T02:03:51.200 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:51.201 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:03:51.201 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-04-19T02:03:51.201 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:51.216 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-19T02:03:51.217 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:51.217+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:51.267 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-04-19T02:03:51.267 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T02:03:51.299 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-04-19T02:03:51.299 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T02:03:51.319 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:51.322+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:51.329 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-04-19T02:03:51.330 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T02:03:51.338 INFO:tasks.rados.rados.0.smithi059.stdout:295: finishing write tid 1 to smithi05930959-41 2024-04-19T02:03:51.338 INFO:tasks.rados.rados.0.smithi059.stdout:295: finishing write tid 2 to smithi05930959-41 2024-04-19T02:03:51.338 INFO:tasks.rados.rados.0.smithi059.stdout:295: finishing write tid 3 to smithi05930959-41 2024-04-19T02:03:51.338 INFO:tasks.rados.rados.0.smithi059.stdout:295: finishing write tid 4 to smithi05930959-41 2024-04-19T02:03:51.338 INFO:tasks.rados.rados.0.smithi059.stdout:295: finishing write tid 5 to smithi05930959-41 2024-04-19T02:03:51.338 INFO:tasks.rados.rados.0.smithi059.stdout:295: finishing write tid 6 to smithi05930959-41 2024-04-19T02:03:51.339 INFO:tasks.rados.rados.0.smithi059.stdout:295: oid 41 updating version 0 to 77 2024-04-19T02:03:51.339 INFO:tasks.rados.rados.0.smithi059.stdout:295: oid 41 updating version 77 to 78 2024-04-19T02:03:51.339 INFO:tasks.rados.rados.0.smithi059.stdout:295: oid 41 version 78 is already newer than 75 2024-04-19T02:03:51.339 INFO:tasks.rados.rados.0.smithi059.stdout:295: oid 41 updating version 78 to 79 2024-04-19T02:03:51.339 INFO:tasks.rados.rados.0.smithi059.stdout:295: oid 41 version 79 is already newer than 76 2024-04-19T02:03:51.339 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 41 v 79 (ObjNum 139 snap 22 seq_num 139) dirty exists 2024-04-19T02:03:51.339 INFO:tasks.rados.rados.0.smithi059.stdout:295: left oid 41 (ObjNum 139 snap 22 seq_num 139) 2024-04-19T02:03:51.350 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-04-19T02:03:51.350 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T02:03:51.371 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-04-19T02:03:51.371 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T02:03:51.375 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:51.375 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:51.390 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-19T02:03:51.392 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-04-19T02:03:51.392 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T02:03:51.413 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-04-19T02:03:51.414 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T02:03:51.419 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:51.421+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:51.434 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-04-19T02:03:51.435 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T02:03:51.456 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-04-19T02:03:51.456 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T02:03:51.477 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-04-19T02:03:51.478 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T02:03:51.500 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-04-19T02:03:51.500 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T02:03:51.506 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:51.506 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:51.521 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-19T02:03:51.521 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:51.521+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:51.523 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-04-19T02:03:51.523 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T02:03:51.622 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:51.625+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:51.639 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:51.639 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:51.654 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-19T02:03:51.722 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:51.722+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:51.775 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:51.775 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:51.790 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-19T02:03:51.823 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:51.825+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:51.924 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:51.925+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:51.958 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:51.958 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:51.973 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-19T02:03:52.025 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:52.025+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:52.045 INFO:teuthology.orchestra.run.smithi059.stderr:set pool 3 pgp_num to 16 2024-04-19T02:03:52.050 INFO:tasks.rados.rados.0.smithi059.stdout:292: done (4 left) 2024-04-19T02:03:52.050 INFO:tasks.rados.rados.0.smithi059.stdout:293: done (3 left) 2024-04-19T02:03:52.050 INFO:tasks.rados.rados.0.smithi059.stdout:294: done (2 left) 2024-04-19T02:03:52.050 INFO:tasks.rados.rados.0.smithi059.stdout:295: done (1 left) 2024-04-19T02:03:52.050 INFO:tasks.rados.rados.0.smithi059.stdout:296: done (0 left) 2024-04-19T02:03:52.050 INFO:tasks.rados.rados.0.smithi059.stdout:297: read oid 3 snap -1 2024-04-19T02:03:52.050 INFO:tasks.rados.rados.0.smithi059.stdout:297: expect (ObjNum 108 snap 15 seq_num 108) 2024-04-19T02:03:52.050 INFO:tasks.rados.rados.0.smithi059.stdout:298: snap_create 2024-04-19T02:03:52.073 INFO:tasks.rados.rados.0.smithi059.stdout:297: expect (ObjNum 108 snap 15 seq_num 108) 2024-04-19T02:03:52.114 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:52.114 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:52.126 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:52.126+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:52.128 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-19T02:03:52.227 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:03:52.229+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:03:52.245 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:52.246 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:52.260 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-19T02:03:52.347 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:03:52.349+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:03:52.425 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:52.425 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:52.428 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:52.429+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:52.439 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-19T02:03:52.530 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:52.530+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:52.607 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:52.607 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:52.622 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-19T02:03:52.630 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:52.633+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:52.731 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:52.733+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:52.738 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:52.738 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:52.752 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-19T02:03:52.832 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:52.834+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:52.876 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:52.876 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:52.890 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-19T02:03:52.934 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:52.937+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:53.013 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:53.013 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:53.027 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-19T02:03:53.035 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:53.038+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:53.053 INFO:tasks.rados.rados.0.smithi059.stdout:297: done (1 left) 2024-04-19T02:03:53.053 INFO:tasks.rados.rados.0.smithi059.stdout:298: done (0 left) 2024-04-19T02:03:53.053 INFO:tasks.rados.rados.0.smithi059.stdout:299: snap_remove snap 22 2024-04-19T02:03:53.083 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-19T02:03:53.136 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:53.138+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:53.147 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:53.147 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:53.162 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-19T02:03:53.237 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:53.237+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:53.280 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:53.280 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:53.295 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-19T02:03:53.338 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:53.341+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:53.416 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:53.416 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:53.430 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-19T02:03:53.438 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:53.441+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:53.539 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:53.541+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:53.543 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T02:03:53.551 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:53.551 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:53.567 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-19T02:03:53.640 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:53.641+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:53.688 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:53.688 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:53.698 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-04-19T02:03:53.699 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T02:03:53.702 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-19T02:03:53.716 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-04-19T02:03:53.716 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T02:03:53.734 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-04-19T02:03:53.734 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T02:03:53.741 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:03:53.741+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:03:53.751 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-04-19T02:03:53.751 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T02:03:53.769 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-04-19T02:03:53.769 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T02:03:53.789 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-04-19T02:03:53.789 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T02:03:53.809 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-04-19T02:03:53.810 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T02:03:53.831 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-04-19T02:03:53.832 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T02:03:53.834 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:53.834 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:53.842 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:53.842+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:53.849 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-19T02:03:53.852 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-04-19T02:03:53.852 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T02:03:53.871 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-04-19T02:03:53.871 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T02:03:53.889 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-04-19T02:03:53.889 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T02:03:53.907 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-04-19T02:03:53.907 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T02:03:53.943 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:53.945+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:53.971 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:53.971 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:03:53.971 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-04-19T02:03:53.971 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-04-19T02:03:53.971 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:53.985 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-19T02:03:54.043 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:54.046+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:54.050 INFO:teuthology.orchestra.run.smithi059.stderr:nodeep-scrub is unset 2024-04-19T02:03:54.055 INFO:tasks.rados.rados.0.smithi059.stdout:299: done (0 left) 2024-04-19T02:03:54.055 INFO:tasks.rados.rados.0.smithi059.stdout:300: read oid 41 snap -1 2024-04-19T02:03:54.055 INFO:tasks.rados.rados.0.smithi059.stdout:300: expect (ObjNum 139 snap 22 seq_num 139) 2024-04-19T02:03:54.055 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 14 current snap is 24 2024-04-19T02:03:54.055 INFO:tasks.rados.rados.0.smithi059.stdout:301: seq_num 140 ranges {0=802816,802816=425984,1228800=720896,1949696=409600} 2024-04-19T02:03:54.059 INFO:tasks.rados.rados.0.smithi059.stdout:301: writing smithi05930959-14 from 0 to 802816 tid 1 2024-04-19T02:03:54.062 INFO:tasks.rados.rados.0.smithi059.stdout:301: writing smithi05930959-14 from 802816 to 1228800 tid 2 2024-04-19T02:03:54.068 INFO:tasks.rados.rados.0.smithi059.stdout:301: writing smithi05930959-14 from 1228800 to 1949696 tid 3 2024-04-19T02:03:54.071 INFO:tasks.rados.rados.0.smithi059.stdout:301: writing smithi05930959-14 from 1949696 to 2359296 tid 4 2024-04-19T02:03:54.071 INFO:tasks.rados.rados.0.smithi059.stdout:302: read oid 11 snap 16 2024-04-19T02:03:54.071 INFO:tasks.rados.rados.0.smithi059.stdout:302: expect deleted 2024-04-19T02:03:54.071 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 28 current snap is 24 2024-04-19T02:03:54.071 INFO:tasks.rados.rados.0.smithi059.stdout:303: seq_num 141 ranges {458752=425984,884736=573440,1458176=425984,1884160=524288,2408448=425984} 2024-04-19T02:03:54.077 INFO:tasks.rados.rados.0.smithi059.stdout:303: writing smithi05930959-28 from 458752 to 884736 tid 1 2024-04-19T02:03:54.081 INFO:tasks.rados.rados.0.smithi059.stdout:303: writing smithi05930959-28 from 884736 to 1458176 tid 2 2024-04-19T02:03:54.084 INFO:tasks.rados.rados.0.smithi059.stdout:303: writing smithi05930959-28 from 1458176 to 1884160 tid 3 2024-04-19T02:03:54.087 INFO:tasks.rados.rados.0.smithi059.stdout:303: writing smithi05930959-28 from 1884160 to 2408448 tid 4 2024-04-19T02:03:54.090 INFO:tasks.rados.rados.0.smithi059.stdout:303: writing smithi05930959-28 from 2408448 to 2834432 tid 5 2024-04-19T02:03:54.091 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 21 current snap is 24 2024-04-19T02:03:54.091 INFO:tasks.rados.rados.0.smithi059.stdout:304: seq_num 142 ranges {1687552=442368,2129920=720896,2850816=540672,3391488=688128} 2024-04-19T02:03:54.105 INFO:tasks.rados.rados.0.smithi059.stdout:304: writing smithi05930959-21 from 1687552 to 2129920 tid 1 2024-04-19T02:03:54.113 INFO:tasks.rados.rados.0.smithi059.stdout:304: writing smithi05930959-21 from 2129920 to 2850816 tid 2 2024-04-19T02:03:54.116 INFO:tasks.rados.rados.0.smithi059.stdout:304: writing smithi05930959-21 from 2850816 to 3391488 tid 3 2024-04-19T02:03:54.121 INFO:tasks.rados.rados.0.smithi059.stdout:304: writing smithi05930959-21 from 3391488 to 4079616 tid 4 2024-04-19T02:03:54.121 INFO:tasks.rados.rados.0.smithi059.stdout:305: delete oid 5 current snap is 24 2024-04-19T02:03:54.121 INFO:tasks.rados.rados.0.smithi059.stdout:301: finishing write tid 1 to smithi05930959-14 2024-04-19T02:03:54.121 INFO:tasks.rados.rados.0.smithi059.stdout:301: finishing write tid 2 to smithi05930959-14 2024-04-19T02:03:54.122 INFO:tasks.rados.rados.0.smithi059.stdout:301: finishing write tid 3 to smithi05930959-14 2024-04-19T02:03:54.122 INFO:tasks.rados.rados.0.smithi059.stdout:301: finishing write tid 4 to smithi05930959-14 2024-04-19T02:03:54.122 INFO:tasks.rados.rados.0.smithi059.stdout:301: finishing write tid 5 to smithi05930959-14 2024-04-19T02:03:54.122 INFO:tasks.rados.rados.0.smithi059.stdout:301: finishing write tid 6 to smithi05930959-14 2024-04-19T02:03:54.122 INFO:tasks.rados.rados.0.smithi059.stdout:301: oid 14 updating version 0 to 77 2024-04-19T02:03:54.122 INFO:tasks.rados.rados.0.smithi059.stdout:301: oid 14 version 77 is already newer than 75 2024-04-19T02:03:54.122 INFO:tasks.rados.rados.0.smithi059.stdout:301: oid 14 updating version 77 to 78 2024-04-19T02:03:54.122 INFO:tasks.rados.rados.0.smithi059.stdout:301: oid 14 version 78 is already newer than 76 2024-04-19T02:03:54.122 INFO:tasks.rados.rados.0.smithi059.stdout:301: oid 14 updating version 78 to 79 2024-04-19T02:03:54.122 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 14 v 79 (ObjNum 140 snap 24 seq_num 140) dirty exists 2024-04-19T02:03:54.122 INFO:tasks.rados.rados.0.smithi059.stdout:301: left oid 14 (ObjNum 140 snap 24 seq_num 140) 2024-04-19T02:03:54.122 INFO:tasks.rados.rados.0.smithi059.stdout:303: finishing write tid 1 to smithi05930959-28 2024-04-19T02:03:54.122 INFO:tasks.rados.rados.0.smithi059.stdout:303: finishing write tid 2 to smithi05930959-28 2024-04-19T02:03:54.122 INFO:tasks.rados.rados.0.smithi059.stdout:303: finishing write tid 3 to smithi05930959-28 2024-04-19T02:03:54.122 INFO:tasks.rados.rados.0.smithi059.stdout:303: finishing write tid 4 to smithi05930959-28 2024-04-19T02:03:54.123 INFO:tasks.rados.rados.0.smithi059.stdout:303: finishing write tid 5 to smithi05930959-28 2024-04-19T02:03:54.123 INFO:tasks.rados.rados.0.smithi059.stdout:303: finishing write tid 6 to smithi05930959-28 2024-04-19T02:03:54.123 INFO:tasks.rados.rados.0.smithi059.stdout:303: finishing write tid 7 to smithi05930959-28 2024-04-19T02:03:54.123 INFO:tasks.rados.rados.0.smithi059.stdout:303: oid 28 updating version 0 to 26 2024-04-19T02:03:54.123 INFO:tasks.rados.rados.0.smithi059.stdout:303: oid 28 updating version 26 to 27 2024-04-19T02:03:54.123 INFO:tasks.rados.rados.0.smithi059.stdout:303: oid 28 version 27 is already newer than 23 2024-04-19T02:03:54.123 INFO:tasks.rados.rados.0.smithi059.stdout:303: oid 28 version 27 is already newer than 25 2024-04-19T02:03:54.123 INFO:tasks.rados.rados.0.smithi059.stdout:303: oid 28 updating version 27 to 28 2024-04-19T02:03:54.123 INFO:tasks.rados.rados.0.smithi059.stdout:303: oid 28 version 28 is already newer than 24 2024-04-19T02:03:54.123 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 28 v 28 (ObjNum 141 snap 24 seq_num 141) dirty exists 2024-04-19T02:03:54.123 INFO:tasks.rados.rados.0.smithi059.stdout:303: left oid 28 (ObjNum 141 snap 24 seq_num 141) 2024-04-19T02:03:54.123 INFO:tasks.rados.rados.0.smithi059.stdout:300: expect (ObjNum 139 snap 22 seq_num 139) 2024-04-19T02:03:54.144 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:54.146+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:54.182 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:54.183 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-04-19T02:03:54.183 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-04-19T02:03:54.183 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-04-19T02:03:54.183 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.183 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:54.183 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:02:59.539411+0000", 2024-04-19T02:03:54.183 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 54.643254888999998, 2024-04-19T02:03:54.183 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.43600685099999997, 2024-04-19T02:03:54.183 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.183 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.183 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:54.183 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.183 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:54.183 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:54.183 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 1 2024-04-19T02:03:54.184 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.184 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.184 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.184 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.184 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539411+0000", 2024-04-19T02:03:54.184 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.184 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.184 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.184 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.184 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539411+0000", 2024-04-19T02:03:54.184 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.184 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.184 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.184 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.184 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539413+0000", 2024-04-19T02:03:54.185 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3970000000000001e-06 2024-04-19T02:03:54.185 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.185 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.185 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.185 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539429+0000", 2024-04-19T02:03:54.185 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5153e-05 2024-04-19T02:03:54.185 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.185 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.185 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.185 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539450+0000", 2024-04-19T02:03:54.185 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1243e-05 2024-04-19T02:03:54.185 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.185 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.185 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.186 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539470+0000", 2024-04-19T02:03:54.186 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0017000000000002e-05 2024-04-19T02:03:54.186 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.186 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.186 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.186 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539605+0000", 2024-04-19T02:03:54.186 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013489200000000001 2024-04-19T02:03:54.186 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.186 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.186 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-04-19T02:03:54.186 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539633+0000", 2024-04-19T02:03:54.186 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8113e-05 2024-04-19T02:03:54.187 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.187 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.187 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.187 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.972415+0000", 2024-04-19T02:03:54.187 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.43278235300000001 2024-04-19T02:03:54.187 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.187 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.187 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.187 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.972942+0000", 2024-04-19T02:03:54.187 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000526379 2024-04-19T02:03:54.187 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.187 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.187 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:54.188 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.973384+0000", 2024-04-19T02:03:54.188 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00044265 2024-04-19T02:03:54.188 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.188 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.188 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:54.188 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.974795+0000", 2024-04-19T02:03:54.188 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001410592 2024-04-19T02:03:54.188 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.188 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.188 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:54.188 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.975249+0000", 2024-04-19T02:03:54.188 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00045376999999999999 2024-04-19T02:03:54.188 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.188 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.188 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:54.189 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.975346+0000", 2024-04-19T02:03:54.189 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7916000000000002e-05 2024-04-19T02:03:54.189 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.189 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.189 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.189 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.975418+0000", 2024-04-19T02:03:54.189 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1376000000000002e-05 2024-04-19T02:03:54.189 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.189 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.189 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.189 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.189 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.189 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:54.189 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.791660+0000", 2024-04-19T02:03:54.189 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 53.391005516, 2024-04-19T02:03:54.189 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021026688000000002, 2024-04-19T02:03:54.190 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.190 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.190 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:54.190 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.190 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:54.190 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:54.190 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 139 2024-04-19T02:03:54.190 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.190 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.190 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.190 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.190 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791660+0000", 2024-04-19T02:03:54.190 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.190 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.191 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.191 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.191 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791660+0000", 2024-04-19T02:03:54.191 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.191 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.191 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.191 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.191 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791661+0000", 2024-04-19T02:03:54.191 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2699999999999999e-07 2024-04-19T02:03:54.191 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.191 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.191 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.191 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791666+0000", 2024-04-19T02:03:54.191 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4160000000000003e-06 2024-04-19T02:03:54.191 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.191 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.192 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.192 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791676+0000", 2024-04-19T02:03:54.192 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1169999999999993e-06 2024-04-19T02:03:54.192 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.192 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.192 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.192 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791683+0000", 2024-04-19T02:03:54.192 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8369999999999997e-06 2024-04-19T02:03:54.192 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.192 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.192 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.192 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.807160+0000", 2024-04-19T02:03:54.192 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.01547656 2024-04-19T02:03:54.192 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.193 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.193 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.193 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.807408+0000", 2024-04-19T02:03:54.193 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000248485 2024-04-19T02:03:54.193 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.193 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.193 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:54.193 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.807622+0000", 2024-04-19T02:03:54.193 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021381199999999999 2024-04-19T02:03:54.193 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.193 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.193 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:54.193 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812397+0000", 2024-04-19T02:03:54.193 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0047750229999999998 2024-04-19T02:03:54.193 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.193 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.194 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:54.194 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812434+0000", 2024-04-19T02:03:54.194 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6869000000000002e-05 2024-04-19T02:03:54.194 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.194 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.194 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:54.194 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812571+0000", 2024-04-19T02:03:54.194 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013649199999999999 2024-04-19T02:03:54.194 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.194 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.194 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.194 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812687+0000", 2024-04-19T02:03:54.194 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011635 2024-04-19T02:03:54.195 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.195 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.195 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.195 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.195 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.195 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:54.195 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.791694+0000", 2024-04-19T02:03:54.195 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 53.390971596, 2024-04-19T02:03:54.195 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021303507999999999, 2024-04-19T02:03:54.195 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.195 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.195 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:54.195 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.195 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:54.195 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:54.195 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 140 2024-04-19T02:03:54.196 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.196 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.196 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.196 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.196 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791694+0000", 2024-04-19T02:03:54.196 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.196 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.196 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.196 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.196 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791694+0000", 2024-04-19T02:03:54.196 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.196 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.196 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.196 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.196 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791695+0000", 2024-04-19T02:03:54.196 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2699999999999999e-07 2024-04-19T02:03:54.197 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.197 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.197 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.197 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791829+0000", 2024-04-19T02:03:54.197 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013436500000000001 2024-04-19T02:03:54.197 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.197 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.197 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.197 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791841+0000", 2024-04-19T02:03:54.197 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1707e-05 2024-04-19T02:03:54.197 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.197 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.197 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.197 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791850+0000", 2024-04-19T02:03:54.198 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5799999999999992e-06 2024-04-19T02:03:54.198 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.198 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.198 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.198 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.808635+0000", 2024-04-19T02:03:54.198 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016785133000000001 2024-04-19T02:03:54.198 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.198 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.198 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.198 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.808862+0000", 2024-04-19T02:03:54.198 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022693799999999999 2024-04-19T02:03:54.198 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.198 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.199 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:54.199 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.809242+0000", 2024-04-19T02:03:54.199 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00038053400000000003 2024-04-19T02:03:54.199 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.199 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.199 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:54.199 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812709+0000", 2024-04-19T02:03:54.199 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0034666929999999999 2024-04-19T02:03:54.199 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.199 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.199 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:54.199 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812796+0000", 2024-04-19T02:03:54.199 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6877000000000002e-05 2024-04-19T02:03:54.199 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.199 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.200 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:54.200 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812910+0000", 2024-04-19T02:03:54.200 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000113929 2024-04-19T02:03:54.200 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.200 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.200 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.200 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812998+0000", 2024-04-19T02:03:54.200 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8024999999999998e-05 2024-04-19T02:03:54.200 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.200 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.200 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.200 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.200 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.200 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:03:54.200 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.791863+0000", 2024-04-19T02:03:54.200 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 53.390802786999998, 2024-04-19T02:03:54.201 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021620092, 2024-04-19T02:03:54.201 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.201 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.201 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:54.201 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.201 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:03:54.201 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:03:54.201 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 141 2024-04-19T02:03:54.201 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.201 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.201 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.201 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.201 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791863+0000", 2024-04-19T02:03:54.201 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.201 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.201 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.202 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.202 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791863+0000", 2024-04-19T02:03:54.202 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.202 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.202 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.202 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.202 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791864+0000", 2024-04-19T02:03:54.202 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0599999999999999e-07 2024-04-19T02:03:54.202 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.202 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.202 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.202 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791871+0000", 2024-04-19T02:03:54.202 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1040000000000001e-06 2024-04-19T02:03:54.202 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.202 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.203 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.203 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791881+0000", 2024-04-19T02:03:54.203 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4329999999999993e-06 2024-04-19T02:03:54.203 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.203 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.203 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.203 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791888+0000", 2024-04-19T02:03:54.203 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7360000000000001e-06 2024-04-19T02:03:54.203 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.203 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.203 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.203 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.811476+0000", 2024-04-19T02:03:54.203 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019588186 2024-04-19T02:03:54.203 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.203 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.204 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.204 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.811595+0000", 2024-04-19T02:03:54.204 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011901200000000001 2024-04-19T02:03:54.204 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.204 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.204 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:03:54.204 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.811793+0000", 2024-04-19T02:03:54.204 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000197958 2024-04-19T02:03:54.204 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.204 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.204 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:03:54.204 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.813066+0000", 2024-04-19T02:03:54.204 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001272884 2024-04-19T02:03:54.204 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.204 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.204 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:03:54.205 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.813356+0000", 2024-04-19T02:03:54.205 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028986399999999997 2024-04-19T02:03:54.205 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.205 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.205 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:54.205 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.813430+0000", 2024-04-19T02:03:54.205 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3716999999999999e-05 2024-04-19T02:03:54.205 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.205 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.205 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.205 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.813483+0000", 2024-04-19T02:03:54.205 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3291999999999997e-05 2024-04-19T02:03:54.205 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.205 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.206 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.206 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.206 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.206 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:363 3.ds0 3:bdf44503:::smithi05930959-38:head [append 0~638976] snapc 9=[9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:54.206 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.746432+0000", 2024-04-19T02:03:54.206 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 27.436234332000002, 2024-04-19T02:03:54.206 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.037983646000000003, 2024-04-19T02:03:54.206 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.206 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.206 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:54.206 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.206 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:54.206 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:54.206 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 363 2024-04-19T02:03:54.207 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.207 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.207 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.207 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.207 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.746432+0000", 2024-04-19T02:03:54.207 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.207 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.207 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.207 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.207 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.746432+0000", 2024-04-19T02:03:54.207 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.207 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.207 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.207 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.207 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.746435+0000", 2024-04-19T02:03:54.208 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3160000000000001e-06 2024-04-19T02:03:54.208 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.208 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.208 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.208 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747630+0000", 2024-04-19T02:03:54.208 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001195371 2024-04-19T02:03:54.208 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.208 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.208 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.208 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747664+0000", 2024-04-19T02:03:54.208 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4190000000000003e-05 2024-04-19T02:03:54.208 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.208 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.208 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.208 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747687+0000", 2024-04-19T02:03:54.208 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2368e-05 2024-04-19T02:03:54.209 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.209 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.209 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.209 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747968+0000", 2024-04-19T02:03:54.209 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028139799999999998 2024-04-19T02:03:54.209 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.209 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.209 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.209 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748247+0000", 2024-04-19T02:03:54.209 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00027916600000000002 2024-04-19T02:03:54.209 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.209 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.209 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:54.209 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753590+0000", 2024-04-19T02:03:54.209 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0053429920000000004 2024-04-19T02:03:54.210 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.210 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.210 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:54.210 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.782450+0000", 2024-04-19T02:03:54.210 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.028859510000000001 2024-04-19T02:03:54.210 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.210 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.210 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:54.210 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.784240+0000", 2024-04-19T02:03:54.210 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001790224 2024-04-19T02:03:54.210 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.210 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.210 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.210 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.784415+0000", 2024-04-19T02:03:54.211 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017511100000000001 2024-04-19T02:03:54.211 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.211 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.211 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.211 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.211 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.211 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.5s4 42/26 ECSubWrite(tid=51, reqid=osd.10.0:51, at_version=42'27, trim_to=0'0, roll_forward_to=39'25))", 2024-04-19T02:03:54.211 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.754979+0000", 2024-04-19T02:03:54.211 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 27.427686679000001, 2024-04-19T02:03:54.211 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.027548537000000001, 2024-04-19T02:03:54.211 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.211 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.211 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:54.211 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.212 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.212 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.212 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.754979+0000", 2024-04-19T02:03:54.212 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.212 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.212 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.212 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.212 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.754979+0000", 2024-04-19T02:03:54.212 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.212 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.212 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.213 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.213 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.754980+0000", 2024-04-19T02:03:54.213 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9999999999999996e-07 2024-04-19T02:03:54.213 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.213 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.213 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.213 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755050+0000", 2024-04-19T02:03:54.213 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9447000000000001e-05 2024-04-19T02:03:54.213 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.213 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.213 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.213 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755077+0000", 2024-04-19T02:03:54.213 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7948000000000001e-05 2024-04-19T02:03:54.213 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.213 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.214 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.214 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755100+0000", 2024-04-19T02:03:54.214 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2714999999999999e-05 2024-04-19T02:03:54.214 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.214 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.214 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.214 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760119+0000", 2024-04-19T02:03:54.214 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0050189800000000001 2024-04-19T02:03:54.214 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.214 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.214 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:54.214 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760142+0000", 2024-04-19T02:03:54.214 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2603999999999999e-05 2024-04-19T02:03:54.214 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.215 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.215 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:54.215 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.782453+0000", 2024-04-19T02:03:54.215 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022311218000000001 2024-04-19T02:03:54.215 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.215 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.215 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.215 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.782528+0000", 2024-04-19T02:03:54.215 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4725e-05 2024-04-19T02:03:54.215 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.215 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.215 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.215 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.215 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.215 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:364 3.ds0 3:bdf44503:::smithi05930959-38:head [append 0~622592] snapc 9=[9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:54.215 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.756160+0000", 2024-04-19T02:03:54.216 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 27.426506337999999, 2024-04-19T02:03:54.216 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.034510458000000001, 2024-04-19T02:03:54.216 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.216 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.216 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:54.216 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.216 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:54.216 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:54.216 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 364 2024-04-19T02:03:54.216 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.216 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.216 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.216 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.216 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.756160+0000", 2024-04-19T02:03:54.216 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.217 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.217 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.217 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.217 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.756160+0000", 2024-04-19T02:03:54.217 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.217 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.217 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.217 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.217 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.756163+0000", 2024-04-19T02:03:54.217 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5130000000000001e-06 2024-04-19T02:03:54.217 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.217 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.217 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.217 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.757299+0000", 2024-04-19T02:03:54.217 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0011355289999999999 2024-04-19T02:03:54.217 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.218 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.218 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.218 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.757340+0000", 2024-04-19T02:03:54.218 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1680000000000001e-05 2024-04-19T02:03:54.218 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.218 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.218 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.218 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.757364+0000", 2024-04-19T02:03:54.218 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4215000000000001e-05 2024-04-19T02:03:54.218 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.218 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.218 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.218 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.763302+0000", 2024-04-19T02:03:54.218 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0059373480000000003 2024-04-19T02:03:54.219 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.219 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.219 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.219 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.768600+0000", 2024-04-19T02:03:54.219 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.005298568 2024-04-19T02:03:54.219 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.219 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.219 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:54.219 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.772526+0000", 2024-04-19T02:03:54.219 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039256450000000002 2024-04-19T02:03:54.219 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.219 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.219 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:54.219 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.790307+0000", 2024-04-19T02:03:54.219 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017781165000000002 2024-04-19T02:03:54.219 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.220 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.220 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:54.220 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.790491+0000", 2024-04-19T02:03:54.220 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000183845 2024-04-19T02:03:54.220 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.220 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.220 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.220 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.790670+0000", 2024-04-19T02:03:54.220 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017895000000000001 2024-04-19T02:03:54.220 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.220 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.220 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.220 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.220 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.221 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:365 3.ds0 3:bdf44503:::smithi05930959-38:head [append 0~147456] snapc 9=[9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:54.221 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.758315+0000", 2024-04-19T02:03:54.221 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 27.424351162000001, 2024-04-19T02:03:54.221 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.040637806999999998, 2024-04-19T02:03:54.221 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.221 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.221 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:54.221 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.221 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:54.221 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:54.221 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 365 2024-04-19T02:03:54.221 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.221 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.221 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.222 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.222 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758315+0000", 2024-04-19T02:03:54.222 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.222 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.222 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.222 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.222 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758315+0000", 2024-04-19T02:03:54.222 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.222 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.222 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.222 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.222 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758317+0000", 2024-04-19T02:03:54.222 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5129999999999999e-06 2024-04-19T02:03:54.223 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.223 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.223 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.223 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758592+0000", 2024-04-19T02:03:54.223 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000275285 2024-04-19T02:03:54.223 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.223 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.223 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.223 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758625+0000", 2024-04-19T02:03:54.223 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2668e-05 2024-04-19T02:03:54.223 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.223 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.223 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.223 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758662+0000", 2024-04-19T02:03:54.223 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7042999999999998e-05 2024-04-19T02:03:54.224 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.224 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.224 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.224 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.784642+0000", 2024-04-19T02:03:54.224 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.025979510000000001 2024-04-19T02:03:54.224 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.224 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.224 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.224 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.785163+0000", 2024-04-19T02:03:54.224 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00052105999999999995 2024-04-19T02:03:54.224 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.224 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.224 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:54.224 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.788827+0000", 2024-04-19T02:03:54.224 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0036638109999999999 2024-04-19T02:03:54.225 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.225 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.225 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:54.225 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.792982+0000", 2024-04-19T02:03:54.225 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0041554829999999997 2024-04-19T02:03:54.225 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.225 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.225 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:54.225 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.798798+0000", 2024-04-19T02:03:54.225 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0058162379999999996 2024-04-19T02:03:54.225 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.225 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.225 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.226 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.798952+0000", 2024-04-19T02:03:54.226 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000154196 2024-04-19T02:03:54.226 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.226 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.226 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.226 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.226 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.226 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:366 3.ds0 3:bdf44503:::smithi05930959-38:head [setxattr _header (56) in=7b] snapc 9=[9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:54.226 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.758698+0000", 2024-04-19T02:03:54.226 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 27.423968235, 2024-04-19T02:03:54.226 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.041876360000000001, 2024-04-19T02:03:54.226 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.226 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.226 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:54.226 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.227 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:54.227 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:54.227 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 366 2024-04-19T02:03:54.227 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.227 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.227 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.227 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.227 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758698+0000", 2024-04-19T02:03:54.227 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.227 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.227 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.227 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.227 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758698+0000", 2024-04-19T02:03:54.227 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.227 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.228 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.228 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.228 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758699+0000", 2024-04-19T02:03:54.228 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8160000000000001e-06 2024-04-19T02:03:54.228 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.228 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.228 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.228 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758711+0000", 2024-04-19T02:03:54.228 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1070000000000001e-05 2024-04-19T02:03:54.228 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.228 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.228 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.228 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758731+0000", 2024-04-19T02:03:54.228 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0650999999999999e-05 2024-04-19T02:03:54.228 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.229 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.229 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.229 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758747+0000", 2024-04-19T02:03:54.229 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5452e-05 2024-04-19T02:03:54.229 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.229 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.229 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.229 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.790875+0000", 2024-04-19T02:03:54.229 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.032128139999999999 2024-04-19T02:03:54.229 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.229 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.229 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.229 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.791060+0000", 2024-04-19T02:03:54.230 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00018499200000000001 2024-04-19T02:03:54.230 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.230 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.230 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:54.230 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.791992+0000", 2024-04-19T02:03:54.230 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00093232600000000003 2024-04-19T02:03:54.230 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.230 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.230 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:54.230 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.796307+0000", 2024-04-19T02:03:54.230 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0043147899999999998 2024-04-19T02:03:54.230 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.230 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.231 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:54.231 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.800345+0000", 2024-04-19T02:03:54.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.004037789 2024-04-19T02:03:54.231 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.231 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.231 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.800574+0000", 2024-04-19T02:03:54.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022933400000000001 2024-04-19T02:03:54.231 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.231 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.231 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.231 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.231 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:367 3.ds0 3:bdf44503:::smithi05930959-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:54.231 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.758769+0000", 2024-04-19T02:03:54.232 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 27.423897171, 2024-04-19T02:03:54.232 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.049021666999999998, 2024-04-19T02:03:54.232 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.232 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.232 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:54.232 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.232 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:54.232 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:54.232 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 367 2024-04-19T02:03:54.232 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.232 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.232 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.232 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.232 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758769+0000", 2024-04-19T02:03:54.233 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.233 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.233 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.233 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.233 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758769+0000", 2024-04-19T02:03:54.233 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.233 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.233 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.233 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.233 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758770+0000", 2024-04-19T02:03:54.233 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5829999999999999e-06 2024-04-19T02:03:54.233 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.233 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.233 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.233 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758794+0000", 2024-04-19T02:03:54.234 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3408e-05 2024-04-19T02:03:54.234 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.234 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.234 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.234 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758815+0000", 2024-04-19T02:03:54.234 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1049000000000001e-05 2024-04-19T02:03:54.234 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.234 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.234 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.234 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758830+0000", 2024-04-19T02:03:54.234 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4975e-05 2024-04-19T02:03:54.234 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.234 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.235 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.235 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.793116+0000", 2024-04-19T02:03:54.235 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.034285806000000002 2024-04-19T02:03:54.235 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.235 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.235 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:54.235 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.793286+0000", 2024-04-19T02:03:54.235 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017076099999999999 2024-04-19T02:03:54.235 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.235 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.235 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.235 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.801267+0000", 2024-04-19T02:03:54.235 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0079808080000000007 2024-04-19T02:03:54.235 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.236 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.236 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.801431+0000", 2024-04-19T02:03:54.236 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016360499999999999 2024-04-19T02:03:54.236 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.236 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.236 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.807790+0000", 2024-04-19T02:03:54.236 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0063596720000000002 2024-04-19T02:03:54.236 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.236 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.236 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.236 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.236 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:692 3.ds0 3:b3d82b95:::smithi05930959-4:head [append 0~622592] snapc 14=[14,13,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-19T02:03:54.237 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:46.447436+0000", 2024-04-19T02:03:54.237 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 7.7352300500000002, 2024-04-19T02:03:54.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.025270186, 2024-04-19T02:03:54.237 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.237 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.237 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:54.237 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.237 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:54.237 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:54.237 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 692 2024-04-19T02:03:54.237 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.237 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.237 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.237 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.238 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.447436+0000", 2024-04-19T02:03:54.238 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.238 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.238 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.238 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.238 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.447436+0000", 2024-04-19T02:03:54.238 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.238 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.238 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.238 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.238 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.447438+0000", 2024-04-19T02:03:54.238 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3769999999999999e-06 2024-04-19T02:03:54.238 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.239 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.239 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.448282+0000", 2024-04-19T02:03:54.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00084384900000000003 2024-04-19T02:03:54.239 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.239 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.239 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.448312+0000", 2024-04-19T02:03:54.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9975e-05 2024-04-19T02:03:54.239 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.239 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.239 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.448333+0000", 2024-04-19T02:03:54.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1049000000000001e-05 2024-04-19T02:03:54.240 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.240 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.240 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.240 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.453182+0000", 2024-04-19T02:03:54.240 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0048494230000000003 2024-04-19T02:03:54.240 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.240 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.240 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.240 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.453430+0000", 2024-04-19T02:03:54.240 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00024780899999999999 2024-04-19T02:03:54.240 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.240 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.240 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:54.241 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.455753+0000", 2024-04-19T02:03:54.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0023230999999999998 2024-04-19T02:03:54.241 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.241 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.241 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:54.241 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.472262+0000", 2024-04-19T02:03:54.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016509113999999998 2024-04-19T02:03:54.241 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.241 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.241 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:54.241 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.472489+0000", 2024-04-19T02:03:54.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022607000000000001 2024-04-19T02:03:54.242 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.242 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.242 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.242 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.472706+0000", 2024-04-19T02:03:54.242 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021741999999999999 2024-04-19T02:03:54.242 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.242 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.242 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.242 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.242 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.242 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:693 3.ds0 3:b3d82b95:::smithi05930959-4:head [append 0~622592] snapc 14=[14,13,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-19T02:03:54.242 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:46.451588+0000", 2024-04-19T02:03:54.242 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 7.7310781459999998, 2024-04-19T02:03:54.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022143603000000001, 2024-04-19T02:03:54.243 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.243 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.243 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:54.243 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.243 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:54.243 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:54.243 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 693 2024-04-19T02:03:54.243 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.243 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.243 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.243 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.243 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.451588+0000", 2024-04-19T02:03:54.244 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.244 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.244 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.244 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.451588+0000", 2024-04-19T02:03:54.244 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.244 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.244 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.244 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.451590+0000", 2024-04-19T02:03:54.244 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7240000000000001e-06 2024-04-19T02:03:54.244 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.245 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.246 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.452642+0000", 2024-04-19T02:03:54.246 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0010513160000000001 2024-04-19T02:03:54.246 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.246 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.246 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.246 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.452672+0000", 2024-04-19T02:03:54.246 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9762000000000001e-05 2024-04-19T02:03:54.246 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.452687+0000", 2024-04-19T02:03:54.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5469e-05 2024-04-19T02:03:54.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.459492+0000", 2024-04-19T02:03:54.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0068048099999999997 2024-04-19T02:03:54.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.459822+0000", 2024-04-19T02:03:54.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00032999800000000002 2024-04-19T02:03:54.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:54.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.462024+0000", 2024-04-19T02:03:54.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0022026319999999999 2024-04-19T02:03:54.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:54.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.472726+0000", 2024-04-19T02:03:54.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010701436999999999 2024-04-19T02:03:54.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:54.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.473503+0000", 2024-04-19T02:03:54.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00077692600000000005 2024-04-19T02:03:54.249 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.249 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.249 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.473731+0000", 2024-04-19T02:03:54.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022852899999999999 2024-04-19T02:03:54.249 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.249 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.249 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.249 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.249 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:694 3.ds0 3:b3d82b95:::smithi05930959-4:head [append 0~311296] snapc 14=[14,13,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-19T02:03:54.249 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:46.453801+0000", 2024-04-19T02:03:54.249 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 7.7288649979999997, 2024-04-19T02:03:54.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.023717372, 2024-04-19T02:03:54.249 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.250 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.250 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:54.250 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.250 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:54.250 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:54.250 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 694 2024-04-19T02:03:54.250 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.250 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.453801+0000", 2024-04-19T02:03:54.250 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.250 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.251 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.251 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.453801+0000", 2024-04-19T02:03:54.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.251 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.251 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.251 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.251 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.453803+0000", 2024-04-19T02:03:54.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5969999999999999e-06 2024-04-19T02:03:54.251 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.251 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.251 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.251 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.454378+0000", 2024-04-19T02:03:54.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00057492100000000005 2024-04-19T02:03:54.251 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.251 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.252 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.252 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.454408+0000", 2024-04-19T02:03:54.252 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0037999999999999e-05 2024-04-19T02:03:54.252 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.252 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.252 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.252 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.454424+0000", 2024-04-19T02:03:54.252 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5177000000000001e-05 2024-04-19T02:03:54.252 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.252 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.252 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.252 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.464961+0000", 2024-04-19T02:03:54.252 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010537187999999999 2024-04-19T02:03:54.252 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.252 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.465414+0000", 2024-04-19T02:03:54.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00045295999999999997 2024-04-19T02:03:54.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:54.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.467274+0000", 2024-04-19T02:03:54.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0018604170000000001 2024-04-19T02:03:54.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:54.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.472730+0000", 2024-04-19T02:03:54.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0054556880000000002 2024-04-19T02:03:54.254 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.254 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.254 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:54.254 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.477358+0000", 2024-04-19T02:03:54.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0046284840000000004 2024-04-19T02:03:54.254 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.254 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.254 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.254 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.477518+0000", 2024-04-19T02:03:54.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015990199999999999 2024-04-19T02:03:54.254 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.254 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.254 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.255 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.255 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:695 3.ds0 3:b3d82b95:::smithi05930959-4:head [setxattr _header (56) in=7b] snapc 14=[14,13,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-19T02:03:54.255 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:46.454454+0000", 2024-04-19T02:03:54.255 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 7.7282117369999996, 2024-04-19T02:03:54.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.024694714999999999, 2024-04-19T02:03:54.255 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.255 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.255 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:54.255 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.255 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:54.255 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:54.255 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 695 2024-04-19T02:03:54.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.256 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.256 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.454454+0000", 2024-04-19T02:03:54.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.256 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.454454+0000", 2024-04-19T02:03:54.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.256 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.454455+0000", 2024-04-19T02:03:54.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2470000000000001e-06 2024-04-19T02:03:54.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.454463+0000", 2024-04-19T02:03:54.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6860000000000006e-06 2024-04-19T02:03:54.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.454477+0000", 2024-04-19T02:03:54.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4192999999999999e-05 2024-04-19T02:03:54.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.454486+0000", 2024-04-19T02:03:54.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2099999999999999e-06 2024-04-19T02:03:54.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.469372+0000", 2024-04-19T02:03:54.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014885116 2024-04-19T02:03:54.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.469684+0000", 2024-04-19T02:03:54.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00031224600000000002 2024-04-19T02:03:54.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.259 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:54.259 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.471030+0000", 2024-04-19T02:03:54.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0013460200000000001 2024-04-19T02:03:54.259 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.259 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:54.259 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.477113+0000", 2024-04-19T02:03:54.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0060836550000000003 2024-04-19T02:03:54.259 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.259 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:54.259 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.478950+0000", 2024-04-19T02:03:54.260 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001836538 2024-04-19T02:03:54.260 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.260 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.260 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.479149+0000", 2024-04-19T02:03:54.260 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00019880399999999999 2024-04-19T02:03:54.260 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.260 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.260 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.260 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.260 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:696 3.ds0 3:b3d82b95:::smithi05930959-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-19T02:03:54.260 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:46.454499+0000", 2024-04-19T02:03:54.260 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 7.7281667160000005, 2024-04-19T02:03:54.260 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.029722386, 2024-04-19T02:03:54.261 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.261 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.261 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:54.261 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.261 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:54.261 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:54.261 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 696 2024-04-19T02:03:54.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.261 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.261 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.261 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.261 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.454499+0000", 2024-04-19T02:03:54.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.454499+0000", 2024-04-19T02:03:54.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.454500+0000", 2024-04-19T02:03:54.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2699999999999998e-07 2024-04-19T02:03:54.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.454504+0000", 2024-04-19T02:03:54.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0729999999999998e-06 2024-04-19T02:03:54.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.454518+0000", 2024-04-19T02:03:54.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3623e-05 2024-04-19T02:03:54.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.454527+0000", 2024-04-19T02:03:54.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8799999999999997e-06 2024-04-19T02:03:54.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.264 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.476941+0000", 2024-04-19T02:03:54.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.02241485 2024-04-19T02:03:54.264 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.264 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.264 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:54.264 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.477102+0000", 2024-04-19T02:03:54.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000160416 2024-04-19T02:03:54.264 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.264 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.264 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.264 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.480818+0000", 2024-04-19T02:03:54.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0037161540000000002 2024-04-19T02:03:54.264 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.264 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.264 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.264 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.481076+0000", 2024-04-19T02:03:54.265 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00025776800000000002 2024-04-19T02:03:54.265 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.265 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.265 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.265 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.484222+0000", 2024-04-19T02:03:54.265 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0031457949999999998 2024-04-19T02:03:54.265 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.265 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.265 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.265 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.265 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.265 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.5s4 67/26 ECSubWrite(tid=85, reqid=client.4509.0:765, at_version=67'41, trim_to=0'0, roll_forward_to=64'39))", 2024-04-19T02:03:54.265 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:51.098386+0000", 2024-04-19T02:03:54.265 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 3.084280213, 2024-04-19T02:03:54.266 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.030421074999999999, 2024-04-19T02:03:54.266 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.266 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.266 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:54.266 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.266 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.266 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.266 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.098386+0000", 2024-04-19T02:03:54.266 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.266 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.266 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.266 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.266 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.098386+0000", 2024-04-19T02:03:54.266 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.098386+0000", 2024-04-19T02:03:54.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9599999999999999e-07 2024-04-19T02:03:54.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.098635+0000", 2024-04-19T02:03:54.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00024883700000000002 2024-04-19T02:03:54.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.098661+0000", 2024-04-19T02:03:54.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5672000000000001e-05 2024-04-19T02:03:54.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.268 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.268 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.098680+0000", 2024-04-19T02:03:54.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8819000000000001e-05 2024-04-19T02:03:54.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.268 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.268 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.098859+0000", 2024-04-19T02:03:54.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017975700000000001 2024-04-19T02:03:54.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.268 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:54.268 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.098911+0000", 2024-04-19T02:03:54.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1198e-05 2024-04-19T02:03:54.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.269 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:54.269 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.128723+0000", 2024-04-19T02:03:54.269 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.029811983 2024-04-19T02:03:54.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.269 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.269 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.128807+0000", 2024-04-19T02:03:54.269 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4112999999999999e-05 2024-04-19T02:03:54.269 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.269 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.269 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.269 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:771 3.ds0 3:b3d82b95:::smithi05930959-4:15 [read 0~3817472,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-04-19T02:03:54.270 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:51.102304+0000", 2024-04-19T02:03:54.270 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 3.0803620829999998, 2024-04-19T02:03:54.270 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021210348, 2024-04-19T02:03:54.270 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.270 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.270 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:54.270 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.270 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:54.270 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:54.270 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 771 2024-04-19T02:03:54.270 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.270 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.270 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.270 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.271 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102304+0000", 2024-04-19T02:03:54.271 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.271 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.271 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.271 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.271 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102304+0000", 2024-04-19T02:03:54.271 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.271 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.271 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.271 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.271 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102308+0000", 2024-04-19T02:03:54.271 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7869999999999999e-06 2024-04-19T02:03:54.271 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.271 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.271 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.272 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102324+0000", 2024-04-19T02:03:54.272 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6218999999999998e-05 2024-04-19T02:03:54.272 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.272 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.272 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.272 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102359+0000", 2024-04-19T02:03:54.272 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5216000000000003e-05 2024-04-19T02:03:54.272 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.272 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.272 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.272 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102381+0000", 2024-04-19T02:03:54.272 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2322000000000001e-05 2024-04-19T02:03:54.272 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.272 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.272 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.273 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102733+0000", 2024-04-19T02:03:54.273 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00035188599999999998 2024-04-19T02:03:54.273 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.273 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.273 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.273 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.103012+0000", 2024-04-19T02:03:54.273 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00027872599999999997 2024-04-19T02:03:54.273 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.273 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.273 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.273 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.123514+0000", 2024-04-19T02:03:54.273 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020502191999999999 2024-04-19T02:03:54.273 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.273 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.274 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.274 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.274 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.274 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:772 3.ds0 3:b3d82b95:::smithi05930959-4:15 [read 0~3817472,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-04-19T02:03:54.274 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:51.102440+0000", 2024-04-19T02:03:54.274 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 3.0802261280000001, 2024-04-19T02:03:54.274 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.025806882, 2024-04-19T02:03:54.274 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.274 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.274 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:54.274 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.274 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:54.274 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:54.274 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 772 2024-04-19T02:03:54.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.275 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.275 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102440+0000", 2024-04-19T02:03:54.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.275 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102440+0000", 2024-04-19T02:03:54.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.275 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102441+0000", 2024-04-19T02:03:54.276 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5200000000000001e-06 2024-04-19T02:03:54.276 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.276 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.276 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.276 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102451+0000", 2024-04-19T02:03:54.276 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3030000000000005e-06 2024-04-19T02:03:54.276 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.276 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.276 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.276 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102473+0000", 2024-04-19T02:03:54.276 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2453e-05 2024-04-19T02:03:54.276 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.276 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.276 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.276 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102489+0000", 2024-04-19T02:03:54.277 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6449000000000001e-05 2024-04-19T02:03:54.277 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.277 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.277 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.277 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.104146+0000", 2024-04-19T02:03:54.277 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001656287 2024-04-19T02:03:54.277 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.277 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.277 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.277 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.104421+0000", 2024-04-19T02:03:54.277 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00027509600000000001 2024-04-19T02:03:54.277 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.277 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.278 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.278 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.128247+0000", 2024-04-19T02:03:54.278 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.023825774000000001 2024-04-19T02:03:54.278 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.278 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.278 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.278 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.278 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.278 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:773 3.ds0 3:b3d82b95:::smithi05930959-4:15 [read 0~3817472,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-04-19T02:03:54.278 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:51.102512+0000", 2024-04-19T02:03:54.278 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 3.080154034, 2024-04-19T02:03:54.278 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.030365781000000001, 2024-04-19T02:03:54.279 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.279 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.279 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:54.279 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.279 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:54.279 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:54.279 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 773 2024-04-19T02:03:54.279 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.279 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.279 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.279 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.279 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102512+0000", 2024-04-19T02:03:54.279 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.279 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.279 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.280 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.280 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102512+0000", 2024-04-19T02:03:54.280 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.280 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.280 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.280 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.280 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102513+0000", 2024-04-19T02:03:54.280 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.347e-06 2024-04-19T02:03:54.280 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.280 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.280 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.280 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102521+0000", 2024-04-19T02:03:54.280 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5429999999999996e-06 2024-04-19T02:03:54.280 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.280 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.281 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102539+0000", 2024-04-19T02:03:54.281 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7876e-05 2024-04-19T02:03:54.281 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.281 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.281 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102553+0000", 2024-04-19T02:03:54.281 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4269e-05 2024-04-19T02:03:54.281 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.281 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.281 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.111639+0000", 2024-04-19T02:03:54.281 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0090859500000000006 2024-04-19T02:03:54.281 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.281 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.282 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.282 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.111958+0000", 2024-04-19T02:03:54.282 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00031888700000000001 2024-04-19T02:03:54.282 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.282 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.282 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.282 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.132878+0000", 2024-04-19T02:03:54.282 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020919909 2024-04-19T02:03:54.282 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.282 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.282 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.282 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.282 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.282 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.5s4 67/26 ECSubWrite(tid=86, reqid=client.4509.0:766, at_version=67'42, trim_to=0'0, roll_forward_to=64'39))", 2024-04-19T02:03:54.282 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:51.105381+0000", 2024-04-19T02:03:54.283 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 3.0772852479999999, 2024-04-19T02:03:54.283 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.027778266999999999, 2024-04-19T02:03:54.283 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.283 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.283 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:54.283 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.283 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.283 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.283 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.105381+0000", 2024-04-19T02:03:54.283 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.283 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.283 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.283 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.283 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.105381+0000", 2024-04-19T02:03:54.283 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.284 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.284 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.284 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.284 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.105383+0000", 2024-04-19T02:03:54.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1900000000000002e-06 2024-04-19T02:03:54.284 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.284 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.284 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.284 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.105745+0000", 2024-04-19T02:03:54.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00036224299999999999 2024-04-19T02:03:54.284 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.284 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.284 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.284 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.105779+0000", 2024-04-19T02:03:54.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4310999999999997e-05 2024-04-19T02:03:54.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.285 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.285 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.285 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.105802+0000", 2024-04-19T02:03:54.285 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2969e-05 2024-04-19T02:03:54.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.285 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.285 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.285 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.110068+0000", 2024-04-19T02:03:54.285 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0042661180000000002 2024-04-19T02:03:54.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.285 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.285 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:54.285 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.110094+0000", 2024-04-19T02:03:54.285 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5814999999999999e-05 2024-04-19T02:03:54.286 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.286 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.286 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:54.286 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.133072+0000", 2024-04-19T02:03:54.286 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022977902000000001 2024-04-19T02:03:54.286 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.286 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.286 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.286 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.133159+0000", 2024-04-19T02:03:54.286 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6718999999999994e-05 2024-04-19T02:03:54.286 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.286 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.286 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.287 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.287 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.287 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:54.288 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-19T02:03:54.289 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:54.289+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:54.389 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:54.389+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:54.457 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:54.457 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:03:54.457 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-04-19T02:03:54.457 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:54.471 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-19T02:03:54.490 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:54.490+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:54.520 INFO:tasks.rados.rados.0.smithi059.stdout:304: finishing write tid 1 to smithi05930959-21 2024-04-19T02:03:54.520 INFO:tasks.rados.rados.0.smithi059.stdout:304: finishing write tid 2 to smithi05930959-21 2024-04-19T02:03:54.520 INFO:tasks.rados.rados.0.smithi059.stdout:304: finishing write tid 3 to smithi05930959-21 2024-04-19T02:03:54.520 INFO:tasks.rados.rados.0.smithi059.stdout:304: finishing write tid 4 to smithi05930959-21 2024-04-19T02:03:54.520 INFO:tasks.rados.rados.0.smithi059.stdout:304: finishing write tid 5 to smithi05930959-21 2024-04-19T02:03:54.520 INFO:tasks.rados.rados.0.smithi059.stdout:304: finishing write tid 6 to smithi05930959-21 2024-04-19T02:03:54.520 INFO:tasks.rados.rados.0.smithi059.stdout:304: oid 21 updating version 0 to 141 2024-04-19T02:03:54.521 INFO:tasks.rados.rados.0.smithi059.stdout:304: oid 21 version 141 is already newer than 140 2024-04-19T02:03:54.521 INFO:tasks.rados.rados.0.smithi059.stdout:304: oid 21 version 141 is already newer than 139 2024-04-19T02:03:54.521 INFO:tasks.rados.rados.0.smithi059.stdout:304: oid 21 version 141 is already newer than 137 2024-04-19T02:03:54.521 INFO:tasks.rados.rados.0.smithi059.stdout:304: oid 21 version 141 is already newer than 138 2024-04-19T02:03:54.521 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 21 v 141 (ObjNum 142 snap 24 seq_num 142) dirty exists 2024-04-19T02:03:54.521 INFO:tasks.rados.rados.0.smithi059.stdout:304: left oid 21 (ObjNum 142 snap 24 seq_num 142) 2024-04-19T02:03:54.521 INFO:tasks.rados.rados.0.smithi059.stdout:300: done (5 left) 2024-04-19T02:03:54.521 INFO:tasks.rados.rados.0.smithi059.stdout:301: done (4 left) 2024-04-19T02:03:54.521 INFO:tasks.rados.rados.0.smithi059.stdout:302: done (3 left) 2024-04-19T02:03:54.521 INFO:tasks.rados.rados.0.smithi059.stdout:303: done (2 left) 2024-04-19T02:03:54.521 INFO:tasks.rados.rados.0.smithi059.stdout:304: done (1 left) 2024-04-19T02:03:54.521 INFO:tasks.rados.rados.0.smithi059.stdout:305: done (0 left) 2024-04-19T02:03:54.521 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 39 current snap is 24 2024-04-19T02:03:54.521 INFO:tasks.rados.rados.0.smithi059.stdout:306: seq_num 144 ranges {3571712=16384} 2024-04-19T02:03:54.542 INFO:tasks.rados.rados.0.smithi059.stdout:306: writing smithi05930959-39 from 3571712 to 3588096 tid 1 2024-04-19T02:03:54.542 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 47 current snap is 24 2024-04-19T02:03:54.542 INFO:tasks.rados.rados.0.smithi059.stdout:307: seq_num 145 ranges {0=32768} 2024-04-19T02:03:54.542 INFO:tasks.rados.rados.0.smithi059.stdout:307: writing smithi05930959-47 from 0 to 32768 tid 1 2024-04-19T02:03:54.542 INFO:tasks.rados.rados.0.smithi059.stdout:308: snap_create 2024-04-19T02:03:54.548 INFO:tasks.rados.rados.0.smithi059.stdout:307: finishing write tid 1 to smithi05930959-47 2024-04-19T02:03:54.549 INFO:tasks.rados.rados.0.smithi059.stdout:307: finishing write tid 2 to smithi05930959-47 2024-04-19T02:03:54.550 INFO:tasks.rados.rados.0.smithi059.stdout:306: finishing write tid 1 to smithi05930959-39 2024-04-19T02:03:54.552 INFO:tasks.rados.rados.0.smithi059.stdout:307: finishing write tid 3 to smithi05930959-47 2024-04-19T02:03:54.552 INFO:tasks.rados.rados.0.smithi059.stdout:307: oid 47 updating version 0 to 66 2024-04-19T02:03:54.552 INFO:tasks.rados.rados.0.smithi059.stdout:307: oid 47 version 66 is already newer than 65 2024-04-19T02:03:54.552 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 47 v 66 (ObjNum 145 snap 24 seq_num 145) dirty exists 2024-04-19T02:03:54.552 INFO:tasks.rados.rados.0.smithi059.stdout:307: left oid 47 (ObjNum 145 snap 24 seq_num 145) 2024-04-19T02:03:54.552 INFO:tasks.rados.rados.0.smithi059.stdout:306: finishing write tid 2 to smithi05930959-39 2024-04-19T02:03:54.555 INFO:tasks.rados.rados.0.smithi059.stdout:306: finishing write tid 3 to smithi05930959-39 2024-04-19T02:03:54.555 INFO:tasks.rados.rados.0.smithi059.stdout:306: oid 39 updating version 0 to 82 2024-04-19T02:03:54.555 INFO:tasks.rados.rados.0.smithi059.stdout:306: oid 39 version 82 is already newer than 81 2024-04-19T02:03:54.555 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 39 v 82 (ObjNum 144 snap 24 seq_num 144) dirty exists 2024-04-19T02:03:54.555 INFO:tasks.rados.rados.0.smithi059.stdout:306: left oid 39 (ObjNum 144 snap 24 seq_num 144) 2024-04-19T02:03:54.591 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:54.593+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:54.635 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:54.635 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:03:54.635 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-04-19T02:03:54.635 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-04-19T02:03:54.635 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:54.649 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-19T02:03:54.692 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:54.694+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:54.774 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:54.774 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-04-19T02:03:54.774 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-04-19T02:03:54.774 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-04-19T02:03:54.774 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.774 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:359 3.cs0 3:3da93e5c:::smithi05930959-42:head [copy-from ver 6] snapc 8=[8,6,5] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:54.774 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:24.774030+0000", 2024-04-19T02:03:54.774 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 29.999696908000001, 2024-04-19T02:03:54.774 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022176986999999999, 2024-04-19T02:03:54.774 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.775 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.775 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:54.775 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.775 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:54.775 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:54.775 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 359 2024-04-19T02:03:54.775 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.775 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.775 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.775 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.775 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774030+0000", 2024-04-19T02:03:54.777 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.777 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.777 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.777 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.777 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774030+0000", 2024-04-19T02:03:54.777 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.777 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.777 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.778 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.778 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774033+0000", 2024-04-19T02:03:54.778 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3859999999999999e-06 2024-04-19T02:03:54.778 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.778 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.778 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.778 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774051+0000", 2024-04-19T02:03:54.778 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8031999999999999e-05 2024-04-19T02:03:54.778 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.778 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.778 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.779 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774086+0000", 2024-04-19T02:03:54.779 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4573999999999998e-05 2024-04-19T02:03:54.779 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.779 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.779 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.779 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774108+0000", 2024-04-19T02:03:54.779 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1931000000000001e-05 2024-04-19T02:03:54.779 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.779 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.779 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.779 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774260+0000", 2024-04-19T02:03:54.779 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015161499999999999 2024-04-19T02:03:54.779 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.780 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.780 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.780 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774532+0000", 2024-04-19T02:03:54.780 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00027218700000000001 2024-04-19T02:03:54.780 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.780 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.780 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:54.780 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.791272+0000", 2024-04-19T02:03:54.780 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016740655 2024-04-19T02:03:54.780 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.780 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.780 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:54.780 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.794135+0000", 2024-04-19T02:03:54.780 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0028628170000000001 2024-04-19T02:03:54.780 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.780 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.781 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:54.781 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.796005+0000", 2024-04-19T02:03:54.781 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0018697 2024-04-19T02:03:54.781 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.781 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.781 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.781 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.796207+0000", 2024-04-19T02:03:54.781 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020209000000000001 2024-04-19T02:03:54.781 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.781 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.781 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.781 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.781 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.781 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:360 3.cs0 3:3da93e5c:::smithi05930959-42:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:03:54.782 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:24.774163+0000", 2024-04-19T02:03:54.782 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 29.999563936000001, 2024-04-19T02:03:54.782 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019807084999999999, 2024-04-19T02:03:54.782 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.782 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.782 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:54.782 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.782 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:54.782 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:54.782 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 360 2024-04-19T02:03:54.782 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.782 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.782 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.782 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.782 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774163+0000", 2024-04-19T02:03:54.783 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.783 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.783 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.783 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.783 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774163+0000", 2024-04-19T02:03:54.783 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.783 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.783 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.783 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.783 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774165+0000", 2024-04-19T02:03:54.783 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.513e-06 2024-04-19T02:03:54.783 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.783 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.783 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.784 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774173+0000", 2024-04-19T02:03:54.784 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.32e-06 2024-04-19T02:03:54.784 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.784 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.784 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.784 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774196+0000", 2024-04-19T02:03:54.784 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3493999999999999e-05 2024-04-19T02:03:54.784 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.784 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.784 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.784 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774213+0000", 2024-04-19T02:03:54.784 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6835999999999999e-05 2024-04-19T02:03:54.784 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.784 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.784 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.784 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.774977+0000", 2024-04-19T02:03:54.785 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00076389999999999997 2024-04-19T02:03:54.785 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.785 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.785 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for blocked object", 2024-04-19T02:03:54.785 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.775221+0000", 2024-04-19T02:03:54.785 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00024397399999999999 2024-04-19T02:03:54.785 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.785 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.785 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.785 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.793417+0000", 2024-04-19T02:03:54.785 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018195896 2024-04-19T02:03:54.785 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.785 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.785 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.785 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.793748+0000", 2024-04-19T02:03:54.786 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000330716 2024-04-19T02:03:54.786 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.786 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.786 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.786 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:24.793970+0000", 2024-04-19T02:03:54.786 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022243600000000001 2024-04-19T02:03:54.786 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.786 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.786 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.786 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.786 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.786 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.ds6 42/26 ECSubWrite(tid=163, reqid=osd.8.0:163, at_version=42'14, trim_to=0'0, roll_forward_to=39'12))", 2024-04-19T02:03:54.786 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.743451+0000", 2024-04-19T02:03:54.786 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 28.030276386000001, 2024-04-19T02:03:54.787 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022917335, 2024-04-19T02:03:54.787 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.787 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.787 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:54.787 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.787 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.787 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.787 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.743451+0000", 2024-04-19T02:03:54.787 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.787 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.787 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.787 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.787 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.743451+0000", 2024-04-19T02:03:54.787 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.787 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.788 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.788 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.788 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.743452+0000", 2024-04-19T02:03:54.788 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.33e-07 2024-04-19T02:03:54.788 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.788 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.788 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.788 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.743513+0000", 2024-04-19T02:03:54.788 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1404999999999999e-05 2024-04-19T02:03:54.788 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.788 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.788 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.788 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.743548+0000", 2024-04-19T02:03:54.788 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5187e-05 2024-04-19T02:03:54.789 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.789 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.789 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.789 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.743573+0000", 2024-04-19T02:03:54.789 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4618e-05 2024-04-19T02:03:54.789 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.789 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.789 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.789 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.743799+0000", 2024-04-19T02:03:54.789 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000226553 2024-04-19T02:03:54.789 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.790 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.790 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:54.790 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.743825+0000", 2024-04-19T02:03:54.790 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6101e-05 2024-04-19T02:03:54.790 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.790 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.790 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:54.790 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.766270+0000", 2024-04-19T02:03:54.790 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022444664999999999 2024-04-19T02:03:54.790 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.790 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.790 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.790 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.766368+0000", 2024-04-19T02:03:54.791 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7873000000000003e-05 2024-04-19T02:03:54.791 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.791 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.791 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.791 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.791 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.791 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s7 42/26 ECSubWrite(tid=132, reqid=osd.3.0:132, at_version=42'63, trim_to=0'0, roll_forward_to=40'61))", 2024-04-19T02:03:54.791 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.747459+0000", 2024-04-19T02:03:54.791 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 28.026268062, 2024-04-19T02:03:54.791 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.023151675, 2024-04-19T02:03:54.791 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.791 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.791 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:54.791 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.791 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.792 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.792 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747459+0000", 2024-04-19T02:03:54.792 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.792 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.792 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.792 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.792 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747459+0000", 2024-04-19T02:03:54.792 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.792 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.792 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.792 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.792 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747460+0000", 2024-04-19T02:03:54.792 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7000000000000004e-07 2024-04-19T02:03:54.793 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.793 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.793 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.793 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747515+0000", 2024-04-19T02:03:54.793 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5248999999999999e-05 2024-04-19T02:03:54.793 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.793 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.793 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.793 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747546+0000", 2024-04-19T02:03:54.793 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1244000000000001e-05 2024-04-19T02:03:54.793 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.793 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.794 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.794 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747569+0000", 2024-04-19T02:03:54.794 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2835000000000001e-05 2024-04-19T02:03:54.794 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.794 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.794 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.794 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747776+0000", 2024-04-19T02:03:54.794 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020755099999999999 2024-04-19T02:03:54.794 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.794 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.794 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:54.794 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747802+0000", 2024-04-19T02:03:54.795 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5694000000000002e-05 2024-04-19T02:03:54.795 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.795 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.795 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:54.795 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.770557+0000", 2024-04-19T02:03:54.795 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022754936 2024-04-19T02:03:54.795 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.795 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.795 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.795 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.770611+0000", 2024-04-19T02:03:54.795 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3495999999999997e-05 2024-04-19T02:03:54.795 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.795 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.795 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.795 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.796 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.796 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.5s3 42/26 ECSubWrite(tid=51, reqid=osd.10.0:51, at_version=42'27, trim_to=0'0, roll_forward_to=39'25))", 2024-04-19T02:03:54.796 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.750076+0000", 2024-04-19T02:03:54.796 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 28.023650711999998, 2024-04-19T02:03:54.796 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020350849000000001, 2024-04-19T02:03:54.796 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.796 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.796 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:54.796 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.796 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.796 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.796 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750076+0000", 2024-04-19T02:03:54.796 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.796 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.796 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.797 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.797 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750076+0000", 2024-04-19T02:03:54.797 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.797 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.797 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.797 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.797 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750077+0000", 2024-04-19T02:03:54.797 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7000000000000004e-07 2024-04-19T02:03:54.797 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.797 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.797 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.797 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750133+0000", 2024-04-19T02:03:54.797 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6119000000000003e-05 2024-04-19T02:03:54.797 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.798 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.798 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.798 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750163+0000", 2024-04-19T02:03:54.798 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0328000000000002e-05 2024-04-19T02:03:54.798 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.798 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.798 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.798 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750220+0000", 2024-04-19T02:03:54.798 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6325e-05 2024-04-19T02:03:54.798 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.798 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.798 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.798 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750376+0000", 2024-04-19T02:03:54.798 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015590100000000001 2024-04-19T02:03:54.799 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.799 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.799 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:54.799 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750401+0000", 2024-04-19T02:03:54.799 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5635000000000001e-05 2024-04-19T02:03:54.799 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.799 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.799 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:54.799 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.770323+0000", 2024-04-19T02:03:54.799 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019921464999999999 2024-04-19T02:03:54.799 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.799 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.799 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.799 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.770427+0000", 2024-04-19T02:03:54.799 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000104306 2024-04-19T02:03:54.800 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.800 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.800 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.800 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.800 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.800 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.as3 42/26 ECSubWrite(tid=82, reqid=osd.11.0:82, at_version=42'34, trim_to=0'0, roll_forward_to=39'32))", 2024-04-19T02:03:54.800 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.750214+0000", 2024-04-19T02:03:54.800 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 28.023512742000001, 2024-04-19T02:03:54.800 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019187731999999999, 2024-04-19T02:03:54.800 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.800 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.800 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:54.800 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.800 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.801 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.801 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750214+0000", 2024-04-19T02:03:54.801 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.801 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.801 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.801 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.801 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750214+0000", 2024-04-19T02:03:54.801 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.801 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.801 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.801 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.801 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750215+0000", 2024-04-19T02:03:54.801 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4000000000000001e-07 2024-04-19T02:03:54.801 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.801 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.802 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.802 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750268+0000", 2024-04-19T02:03:54.802 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3389e-05 2024-04-19T02:03:54.802 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.802 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.802 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.802 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750299+0000", 2024-04-19T02:03:54.802 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0914000000000003e-05 2024-04-19T02:03:54.802 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.802 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.802 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.802 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750322+0000", 2024-04-19T02:03:54.802 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2575e-05 2024-04-19T02:03:54.803 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.803 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.803 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.803 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750578+0000", 2024-04-19T02:03:54.803 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00025623300000000002 2024-04-19T02:03:54.803 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.803 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.803 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:54.803 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750606+0000", 2024-04-19T02:03:54.803 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7568000000000001e-05 2024-04-19T02:03:54.803 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.803 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.803 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:54.803 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.769319+0000", 2024-04-19T02:03:54.804 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018712909 2024-04-19T02:03:54.804 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.804 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.804 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.804 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.769402+0000", 2024-04-19T02:03:54.804 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3404000000000003e-05 2024-04-19T02:03:54.804 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.804 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.804 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.804 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.804 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.804 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s7 42/26 ECSubWrite(tid=134, reqid=osd.3.0:134, at_version=42'65, trim_to=0'0, roll_forward_to=40'61))", 2024-04-19T02:03:54.804 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.750993+0000", 2024-04-19T02:03:54.804 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 28.02273357, 2024-04-19T02:03:54.805 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021026969, 2024-04-19T02:03:54.805 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.805 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.805 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:54.805 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.805 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.805 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.805 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750993+0000", 2024-04-19T02:03:54.805 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.805 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.805 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.805 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.805 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750993+0000", 2024-04-19T02:03:54.805 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.805 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.806 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.806 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.806 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.750995+0000", 2024-04-19T02:03:54.806 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.883e-06 2024-04-19T02:03:54.806 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.806 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.806 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.806 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751070+0000", 2024-04-19T02:03:54.806 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4981000000000006e-05 2024-04-19T02:03:54.806 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.806 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.806 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.806 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751100+0000", 2024-04-19T02:03:54.806 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9944999999999999e-05 2024-04-19T02:03:54.806 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.807 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.807 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.807 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751122+0000", 2024-04-19T02:03:54.807 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1810999999999998e-05 2024-04-19T02:03:54.807 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.807 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.807 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.807 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.762958+0000", 2024-04-19T02:03:54.807 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011836187 2024-04-19T02:03:54.807 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.807 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.807 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:54.807 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.762984+0000", 2024-04-19T02:03:54.807 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5420999999999999e-05 2024-04-19T02:03:54.807 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.808 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.808 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:54.808 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.771936+0000", 2024-04-19T02:03:54.808 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0089524319999999998 2024-04-19T02:03:54.808 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.808 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.808 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.808 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.772020+0000", 2024-04-19T02:03:54.808 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4308999999999995e-05 2024-04-19T02:03:54.808 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.808 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.808 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.808 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.808 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.808 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:377 3.cs0 3:3da93e5c:::smithi05930959-42:head [read 0~589824,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:54.809 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.759004+0000", 2024-04-19T02:03:54.809 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 28.014723055000001, 2024-04-19T02:03:54.809 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.035358317, 2024-04-19T02:03:54.809 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.809 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.809 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:54.809 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.809 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:54.809 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:54.809 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 377 2024-04-19T02:03:54.809 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.809 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.809 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.809 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.809 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759004+0000", 2024-04-19T02:03:54.810 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.810 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.810 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.810 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.810 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759004+0000", 2024-04-19T02:03:54.810 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.810 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.810 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.810 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.810 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759008+0000", 2024-04-19T02:03:54.810 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9190000000000001e-06 2024-04-19T02:03:54.810 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.810 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.810 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.810 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759024+0000", 2024-04-19T02:03:54.811 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6195e-05 2024-04-19T02:03:54.811 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.811 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.811 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.811 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759058+0000", 2024-04-19T02:03:54.811 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4238e-05 2024-04-19T02:03:54.811 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.811 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.811 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.811 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759080+0000", 2024-04-19T02:03:54.811 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1237999999999999e-05 2024-04-19T02:03:54.811 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.811 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.811 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.811 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.771701+0000", 2024-04-19T02:03:54.812 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012621098000000001 2024-04-19T02:03:54.812 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.812 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.812 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:54.812 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.771949+0000", 2024-04-19T02:03:54.812 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00024814599999999998 2024-04-19T02:03:54.812 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.812 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.812 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.812 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.788299+0000", 2024-04-19T02:03:54.812 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016350138 2024-04-19T02:03:54.812 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.812 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.813 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.813 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.788497+0000", 2024-04-19T02:03:54.813 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00019762100000000001 2024-04-19T02:03:54.813 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.813 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.813 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.813 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.794362+0000", 2024-04-19T02:03:54.813 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0058657240000000001 2024-04-19T02:03:54.813 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.813 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.813 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.813 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.813 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.813 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:378 3.cs0 3:3da93e5c:::smithi05930959-42:head [read 0~589824,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:54.813 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.759134+0000", 2024-04-19T02:03:54.814 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 28.014593252000001, 2024-04-19T02:03:54.814 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.044044933000000001, 2024-04-19T02:03:54.814 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.814 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.814 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:54.814 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.814 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:54.814 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:54.814 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 378 2024-04-19T02:03:54.814 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.814 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.814 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.814 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.814 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759134+0000", 2024-04-19T02:03:54.815 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.815 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.815 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.815 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.815 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759134+0000", 2024-04-19T02:03:54.815 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.815 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.815 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.815 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.815 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759136+0000", 2024-04-19T02:03:54.815 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8029999999999999e-06 2024-04-19T02:03:54.815 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.815 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.815 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.815 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759145+0000", 2024-04-19T02:03:54.816 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5789999999999993e-06 2024-04-19T02:03:54.816 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.816 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.816 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.816 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759166+0000", 2024-04-19T02:03:54.816 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0815e-05 2024-04-19T02:03:54.816 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.816 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.816 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.816 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759187+0000", 2024-04-19T02:03:54.816 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1299000000000001e-05 2024-04-19T02:03:54.816 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.816 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.816 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.816 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.772621+0000", 2024-04-19T02:03:54.816 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013434032 2024-04-19T02:03:54.816 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.817 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.817 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:54.817 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.772918+0000", 2024-04-19T02:03:54.817 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00029657400000000002 2024-04-19T02:03:54.817 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.817 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.817 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.817 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.794802+0000", 2024-04-19T02:03:54.817 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021883785999999999 2024-04-19T02:03:54.817 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.817 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.817 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.817 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.795248+0000", 2024-04-19T02:03:54.817 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00044615099999999999 2024-04-19T02:03:54.818 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.818 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.818 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.818 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.803179+0000", 2024-04-19T02:03:54.818 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0079308940000000008 2024-04-19T02:03:54.818 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.818 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.818 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.818 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.818 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.818 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:379 3.cs0 3:3da93e5c:::smithi05930959-42:head [read 0~589824,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:54.818 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.759209+0000", 2024-04-19T02:03:54.818 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 28.014517515000001, 2024-04-19T02:03:54.819 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.045345979000000002, 2024-04-19T02:03:54.819 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.819 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.819 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:54.819 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.819 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:54.819 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:54.819 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 379 2024-04-19T02:03:54.819 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.819 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.819 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.819 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.819 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759209+0000", 2024-04-19T02:03:54.819 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.819 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.820 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.820 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.820 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759209+0000", 2024-04-19T02:03:54.820 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.820 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.820 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.820 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.820 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759210+0000", 2024-04-19T02:03:54.820 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9700000000000005e-07 2024-04-19T02:03:54.820 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.820 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.820 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.820 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759218+0000", 2024-04-19T02:03:54.820 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9589999999999995e-06 2024-04-19T02:03:54.820 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.820 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.821 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.821 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759236+0000", 2024-04-19T02:03:54.821 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7209000000000001e-05 2024-04-19T02:03:54.821 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.821 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.821 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.821 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.759249+0000", 2024-04-19T02:03:54.821 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3956e-05 2024-04-19T02:03:54.821 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.821 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.821 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.821 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.773037+0000", 2024-04-19T02:03:54.821 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013787415000000001 2024-04-19T02:03:54.821 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.821 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.821 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:54.822 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.773251+0000", 2024-04-19T02:03:54.822 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021379099999999999 2024-04-19T02:03:54.822 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.822 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.822 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.822 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.797127+0000", 2024-04-19T02:03:54.822 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.023876128999999999 2024-04-19T02:03:54.822 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.822 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.822 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.822 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.797749+0000", 2024-04-19T02:03:54.822 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00062207600000000005 2024-04-19T02:03:54.822 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.823 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.823 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.823 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.804555+0000", 2024-04-19T02:03:54.823 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0068065469999999996 2024-04-19T02:03:54.823 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.823 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.823 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.823 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.823 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.823 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:529 3.cs0 3:30e59ecb:::smithi05930959-25:head [append 0~540672] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-19T02:03:54.823 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:36.128428+0000", 2024-04-19T02:03:54.823 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 18.645298834999998, 2024-04-19T02:03:54.823 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.01918332, 2024-04-19T02:03:54.823 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.823 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.824 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:54.824 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.824 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:54.824 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:54.824 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 529 2024-04-19T02:03:54.824 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.824 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.824 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.824 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.824 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.128428+0000", 2024-04-19T02:03:54.824 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.824 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.824 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.824 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.825 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.128428+0000", 2024-04-19T02:03:54.825 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.825 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.825 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.825 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.825 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.128431+0000", 2024-04-19T02:03:54.825 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1729999999999999e-06 2024-04-19T02:03:54.825 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.825 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.825 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.825 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.129277+0000", 2024-04-19T02:03:54.825 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00084618699999999998 2024-04-19T02:03:54.825 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.825 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.825 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.825 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.129314+0000", 2024-04-19T02:03:54.826 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6078000000000003e-05 2024-04-19T02:03:54.826 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.826 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.826 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.826 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.129331+0000", 2024-04-19T02:03:54.826 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7592e-05 2024-04-19T02:03:54.826 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.826 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.826 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.826 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.129471+0000", 2024-04-19T02:03:54.826 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014026800000000001 2024-04-19T02:03:54.826 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.826 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.826 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op must wait for map", 2024-04-19T02:03:54.827 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.129484+0000", 2024-04-19T02:03:54.827 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3013000000000001e-05 2024-04-19T02:03:54.827 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.827 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.827 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.827 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.133178+0000", 2024-04-19T02:03:54.827 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00369382 2024-04-19T02:03:54.827 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.827 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.827 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.827 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.133333+0000", 2024-04-19T02:03:54.827 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015457099999999999 2024-04-19T02:03:54.827 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.827 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.828 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:54.828 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.135732+0000", 2024-04-19T02:03:54.828 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002399319 2024-04-19T02:03:54.828 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.828 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.828 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:54.828 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.146340+0000", 2024-04-19T02:03:54.828 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010607441 2024-04-19T02:03:54.828 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.828 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.828 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:54.828 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.147453+0000", 2024-04-19T02:03:54.828 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0011134109999999999 2024-04-19T02:03:54.828 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.828 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.829 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.829 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.147611+0000", 2024-04-19T02:03:54.829 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015844700000000001 2024-04-19T02:03:54.829 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.829 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.829 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.829 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.829 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.829 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:530 3.cs0 3:30e59ecb:::smithi05930959-25:head [append 0~507904] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-19T02:03:54.829 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:36.132638+0000", 2024-04-19T02:03:54.829 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 18.641089429000001, 2024-04-19T02:03:54.829 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015395209, 2024-04-19T02:03:54.829 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.829 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.829 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:54.829 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.830 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:54.830 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:54.830 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 530 2024-04-19T02:03:54.830 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.830 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.830 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.830 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.830 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.132638+0000", 2024-04-19T02:03:54.830 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.830 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.830 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.830 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.830 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.132638+0000", 2024-04-19T02:03:54.830 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.831 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.831 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.831 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.831 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.132640+0000", 2024-04-19T02:03:54.831 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.706e-06 2024-04-19T02:03:54.831 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.831 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.831 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.831 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.133417+0000", 2024-04-19T02:03:54.831 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000776675 2024-04-19T02:03:54.831 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.831 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.831 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.831 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.133440+0000", 2024-04-19T02:03:54.831 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3317999999999999e-05 2024-04-19T02:03:54.831 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.832 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.832 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.832 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.133454+0000", 2024-04-19T02:03:54.832 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4226000000000001e-05 2024-04-19T02:03:54.832 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.832 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.832 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.832 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137804+0000", 2024-04-19T02:03:54.832 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.004349655 2024-04-19T02:03:54.832 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.832 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.832 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.832 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.138121+0000", 2024-04-19T02:03:54.832 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00031658800000000003 2024-04-19T02:03:54.833 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.833 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.833 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:54.833 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.140836+0000", 2024-04-19T02:03:54.833 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0027152410000000002 2024-04-19T02:03:54.833 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.833 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.833 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:54.833 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.146561+0000", 2024-04-19T02:03:54.833 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0057252099999999997 2024-04-19T02:03:54.833 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.833 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.833 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:54.833 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.147916+0000", 2024-04-19T02:03:54.833 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0013549269999999999 2024-04-19T02:03:54.834 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.834 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.834 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.834 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.148033+0000", 2024-04-19T02:03:54.834 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000116663 2024-04-19T02:03:54.834 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.834 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.834 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.834 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.834 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.834 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:531 3.cs0 3:30e59ecb:::smithi05930959-25:head [append 0~622592] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-19T02:03:54.834 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:36.137197+0000", 2024-04-19T02:03:54.834 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 18.63652965, 2024-04-19T02:03:54.835 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015818477000000001, 2024-04-19T02:03:54.835 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.835 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.835 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:54.835 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.835 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:54.835 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:54.835 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 531 2024-04-19T02:03:54.835 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.835 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.835 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.835 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.835 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137197+0000", 2024-04-19T02:03:54.835 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.835 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.835 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.836 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.836 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137197+0000", 2024-04-19T02:03:54.836 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.836 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.836 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.836 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.836 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137199+0000", 2024-04-19T02:03:54.836 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9460000000000001e-06 2024-04-19T02:03:54.836 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.836 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.836 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.836 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.138277+0000", 2024-04-19T02:03:54.836 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001077464 2024-04-19T02:03:54.836 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.836 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.836 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.837 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.138336+0000", 2024-04-19T02:03:54.837 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9528999999999999e-05 2024-04-19T02:03:54.837 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.837 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.837 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.837 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.138360+0000", 2024-04-19T02:03:54.837 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3813000000000001e-05 2024-04-19T02:03:54.837 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.837 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.837 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.837 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.141872+0000", 2024-04-19T02:03:54.838 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0035116739999999998 2024-04-19T02:03:54.838 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.838 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.838 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.838 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.142148+0000", 2024-04-19T02:03:54.838 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00027594000000000002 2024-04-19T02:03:54.838 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.838 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.838 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:54.838 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.144745+0000", 2024-04-19T02:03:54.838 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0025976710000000002 2024-04-19T02:03:54.838 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.838 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.838 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:54.839 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.147177+0000", 2024-04-19T02:03:54.839 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002431164 2024-04-19T02:03:54.839 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.839 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.839 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:54.839 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.152898+0000", 2024-04-19T02:03:54.839 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0057210899999999999 2024-04-19T02:03:54.839 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.839 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.839 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.839 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.153016+0000", 2024-04-19T02:03:54.839 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000118186 2024-04-19T02:03:54.839 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.840 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.840 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.840 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.840 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.840 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:532 3.cs0 3:30e59ecb:::smithi05930959-25:head [append 0~32768] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-19T02:03:54.840 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:36.138403+0000", 2024-04-19T02:03:54.840 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 18.635324064999999, 2024-04-19T02:03:54.840 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015740363, 2024-04-19T02:03:54.840 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.840 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.840 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:54.840 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.840 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:54.840 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:54.840 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 532 2024-04-19T02:03:54.841 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.841 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.841 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.841 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.841 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.138403+0000", 2024-04-19T02:03:54.841 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.841 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.841 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.841 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.841 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.138403+0000", 2024-04-19T02:03:54.841 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.841 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.841 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.841 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.842 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.138405+0000", 2024-04-19T02:03:54.842 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.717e-06 2024-04-19T02:03:54.842 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.842 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.842 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.842 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.138490+0000", 2024-04-19T02:03:54.842 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5657999999999999e-05 2024-04-19T02:03:54.842 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.842 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.842 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.842 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.138515+0000", 2024-04-19T02:03:54.842 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4936e-05 2024-04-19T02:03:54.842 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.842 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.842 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.843 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.138533+0000", 2024-04-19T02:03:54.843 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7609e-05 2024-04-19T02:03:54.843 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.843 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.843 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.843 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.148398+0000", 2024-04-19T02:03:54.843 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.009865548 2024-04-19T02:03:54.843 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.843 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.843 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.843 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.148552+0000", 2024-04-19T02:03:54.843 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015343400000000001 2024-04-19T02:03:54.843 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.843 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.844 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:54.844 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.149528+0000", 2024-04-19T02:03:54.844 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00097626199999999998 2024-04-19T02:03:54.844 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.844 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.844 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:54.844 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.153269+0000", 2024-04-19T02:03:54.844 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00374049 2024-04-19T02:03:54.844 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.844 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.844 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:54.844 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.154020+0000", 2024-04-19T02:03:54.844 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00075099599999999998 2024-04-19T02:03:54.844 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.845 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.845 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.845 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.154143+0000", 2024-04-19T02:03:54.845 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012371300000000001 2024-04-19T02:03:54.845 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.845 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.845 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.845 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.845 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.845 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:533 3.cs0 3:30e59ecb:::smithi05930959-25:head [setxattr _header (57) in=7b] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-19T02:03:54.845 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:36.138978+0000", 2024-04-19T02:03:54.845 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 18.634749241000002, 2024-04-19T02:03:54.845 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016411139000000002, 2024-04-19T02:03:54.845 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.846 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.846 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:54.846 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.846 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:54.846 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:54.846 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 533 2024-04-19T02:03:54.846 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.846 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.846 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.846 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.846 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.138978+0000", 2024-04-19T02:03:54.846 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.846 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.846 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.847 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.847 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.138978+0000", 2024-04-19T02:03:54.847 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.847 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.847 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.847 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.847 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.138980+0000", 2024-04-19T02:03:54.847 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9929999999999998e-06 2024-04-19T02:03:54.847 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.847 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.847 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.847 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.138992+0000", 2024-04-19T02:03:54.847 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1885999999999999e-05 2024-04-19T02:03:54.847 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.848 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.848 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.848 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.139034+0000", 2024-04-19T02:03:54.848 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2700000000000001e-05 2024-04-19T02:03:54.848 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.848 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.848 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.848 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.139056+0000", 2024-04-19T02:03:54.848 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1656000000000001e-05 2024-04-19T02:03:54.848 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.849 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.849 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.849 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.150346+0000", 2024-04-19T02:03:54.849 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011289954 2024-04-19T02:03:54.849 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.849 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.849 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.849 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.150603+0000", 2024-04-19T02:03:54.849 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00025736500000000001 2024-04-19T02:03:54.849 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.849 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.849 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:54.849 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.151787+0000", 2024-04-19T02:03:54.849 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001183293 2024-04-19T02:03:54.849 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.850 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.850 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:54.850 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.153572+0000", 2024-04-19T02:03:54.850 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0017849319999999999 2024-04-19T02:03:54.850 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.850 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.850 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:54.850 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.155167+0000", 2024-04-19T02:03:54.850 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00159585 2024-04-19T02:03:54.850 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.850 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.850 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.850 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.155389+0000", 2024-04-19T02:03:54.850 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022151000000000001 2024-04-19T02:03:54.851 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.851 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.851 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.851 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.851 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.851 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:534 3.cs0 3:30e59ecb:::smithi05930959-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-04-19T02:03:54.851 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:36.139090+0000", 2024-04-19T02:03:54.851 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 18.634637411, 2024-04-19T02:03:54.851 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021395290000000001, 2024-04-19T02:03:54.851 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.851 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.851 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:54.851 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.851 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:54.851 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:54.852 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 534 2024-04-19T02:03:54.852 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.852 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.852 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.852 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.852 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.139090+0000", 2024-04-19T02:03:54.852 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.852 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.852 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.852 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.852 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.139090+0000", 2024-04-19T02:03:54.852 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.852 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.852 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.852 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.853 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.139091+0000", 2024-04-19T02:03:54.853 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.553e-06 2024-04-19T02:03:54.853 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.853 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.853 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.853 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.139100+0000", 2024-04-19T02:03:54.853 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6829999999999997e-06 2024-04-19T02:03:54.853 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.853 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.853 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.853 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.139121+0000", 2024-04-19T02:03:54.853 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0842000000000001e-05 2024-04-19T02:03:54.853 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.853 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.853 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.854 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.139137+0000", 2024-04-19T02:03:54.854 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6735000000000002e-05 2024-04-19T02:03:54.854 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.854 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.854 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.854 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.156383+0000", 2024-04-19T02:03:54.854 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017245607 2024-04-19T02:03:54.854 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.854 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.854 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.854 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.156738+0000", 2024-04-19T02:03:54.854 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00035512299999999999 2024-04-19T02:03:54.854 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.854 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.854 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.855 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.160485+0000", 2024-04-19T02:03:54.855 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0037467469999999999 2024-04-19T02:03:54.855 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.855 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.855 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.855 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.855 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.855 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.as3 57/26 ECSubWrite(tid=136, reqid=osd.11.0:136, at_version=57'62, trim_to=0'0, roll_forward_to=55'60))", 2024-04-19T02:03:54.855 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:41.154863+0000", 2024-04-19T02:03:54.855 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 13.618864122, 2024-04-19T02:03:54.855 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017918618000000001, 2024-04-19T02:03:54.855 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.855 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.855 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:54.855 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.856 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.856 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.856 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:41.154863+0000", 2024-04-19T02:03:54.856 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.856 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.856 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.856 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.856 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:41.154863+0000", 2024-04-19T02:03:54.856 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.856 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.856 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.856 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.856 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:41.154863+0000", 2024-04-19T02:03:54.856 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9699999999999996e-07 2024-04-19T02:03:54.856 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.856 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.857 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.857 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:41.154935+0000", 2024-04-19T02:03:54.857 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1478999999999998e-05 2024-04-19T02:03:54.857 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.857 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.857 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.857 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:41.154964+0000", 2024-04-19T02:03:54.857 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9119e-05 2024-04-19T02:03:54.857 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.857 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.857 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.857 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:41.154987+0000", 2024-04-19T02:03:54.857 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2481999999999999e-05 2024-04-19T02:03:54.858 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.858 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.858 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.858 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:41.155210+0000", 2024-04-19T02:03:54.858 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022362900000000001 2024-04-19T02:03:54.858 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.858 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.858 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:54.858 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:41.155237+0000", 2024-04-19T02:03:54.858 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7282000000000001e-05 2024-04-19T02:03:54.858 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.858 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.858 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:54.858 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:41.172595+0000", 2024-04-19T02:03:54.859 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017357823000000001 2024-04-19T02:03:54.859 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.859 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.859 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.859 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:41.172781+0000", 2024-04-19T02:03:54.859 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000186207 2024-04-19T02:03:54.859 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.859 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.859 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.859 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.859 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.859 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:679 3.cs0 3:30e59ecb:::smithi05930959-25:head [setxattr _header (57) in=7b] snapc 14=[14,13,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-19T02:03:54.859 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:46.375559+0000", 2024-04-19T02:03:54.859 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 8.3981684120000004, 2024-04-19T02:03:54.859 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015915751999999998, 2024-04-19T02:03:54.860 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.860 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.860 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:54.860 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.860 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:54.860 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:54.860 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 679 2024-04-19T02:03:54.860 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.860 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.860 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.860 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.860 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.375559+0000", 2024-04-19T02:03:54.860 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.861 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.861 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.861 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.861 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.375559+0000", 2024-04-19T02:03:54.861 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.861 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.861 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.861 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.861 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.375559+0000", 2024-04-19T02:03:54.861 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0699999999999996e-07 2024-04-19T02:03:54.861 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.861 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.861 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.861 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.375564+0000", 2024-04-19T02:03:54.861 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8029999999999996e-06 2024-04-19T02:03:54.861 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.862 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.862 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.862 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.375582+0000", 2024-04-19T02:03:54.862 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8289000000000001e-05 2024-04-19T02:03:54.862 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.862 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.862 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.862 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.375593+0000", 2024-04-19T02:03:54.862 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0135999999999999e-05 2024-04-19T02:03:54.862 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.862 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.862 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.862 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.387462+0000", 2024-04-19T02:03:54.862 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.01186952 2024-04-19T02:03:54.863 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.863 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.863 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.863 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.387756+0000", 2024-04-19T02:03:54.863 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00029350999999999997 2024-04-19T02:03:54.863 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.863 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.863 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:54.863 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.389067+0000", 2024-04-19T02:03:54.863 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001311368 2024-04-19T02:03:54.863 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.863 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.863 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:54.863 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.390511+0000", 2024-04-19T02:03:54.863 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0014436220000000001 2024-04-19T02:03:54.863 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.864 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.864 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:54.864 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.391304+0000", 2024-04-19T02:03:54.864 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00079366400000000002 2024-04-19T02:03:54.864 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.864 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.864 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.864 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.391474+0000", 2024-04-19T02:03:54.864 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017003299999999999 2024-04-19T02:03:54.864 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.864 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.864 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.864 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.864 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.864 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:680 3.cs0 3:30e59ecb:::smithi05930959-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-19T02:03:54.865 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:46.375603+0000", 2024-04-19T02:03:54.865 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 8.3981237409999991, 2024-04-19T02:03:54.865 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018759568000000001, 2024-04-19T02:03:54.865 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.865 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.865 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:54.865 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.865 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:54.865 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:54.865 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 680 2024-04-19T02:03:54.865 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.865 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.866 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.866 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.866 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.375603+0000", 2024-04-19T02:03:54.866 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.866 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.866 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.866 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.866 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.375603+0000", 2024-04-19T02:03:54.866 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.866 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.866 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.866 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.866 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.375604+0000", 2024-04-19T02:03:54.866 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8999999999999996e-07 2024-04-19T02:03:54.866 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.867 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.867 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.867 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.375607+0000", 2024-04-19T02:03:54.867 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4539999999999999e-06 2024-04-19T02:03:54.867 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.867 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.867 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.867 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.375618+0000", 2024-04-19T02:03:54.867 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0182e-05 2024-04-19T02:03:54.867 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.867 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.867 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.867 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.375626+0000", 2024-04-19T02:03:54.867 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1529999999999997e-06 2024-04-19T02:03:54.867 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.868 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.868 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.868 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.389680+0000", 2024-04-19T02:03:54.868 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014054362000000001 2024-04-19T02:03:54.868 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.868 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.868 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:54.868 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.389834+0000", 2024-04-19T02:03:54.868 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015358 2024-04-19T02:03:54.868 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.868 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.868 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.868 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.392149+0000", 2024-04-19T02:03:54.868 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0023151059999999999 2024-04-19T02:03:54.868 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.869 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.869 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.869 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.392284+0000", 2024-04-19T02:03:54.869 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013507699999999999 2024-04-19T02:03:54.869 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.869 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.869 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.869 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.394363+0000", 2024-04-19T02:03:54.869 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002078964 2024-04-19T02:03:54.869 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.869 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.869 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.869 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.869 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.869 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:758 3.cs0 3:33a480b1:::smithi05930959-6:head [append 0~753664] snapc 16=[16,15,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-19T02:03:54.869 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:50.397461+0000", 2024-04-19T02:03:54.870 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4.3762655559999999, 2024-04-19T02:03:54.870 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017102902, 2024-04-19T02:03:54.870 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:54.870 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:54.870 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:54.870 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:54.870 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:54.870 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:54.870 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 758 2024-04-19T02:03:54.870 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.870 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:54.870 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.870 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:54.871 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:50.397461+0000", 2024-04-19T02:03:54.871 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.871 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.871 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.871 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:54.871 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:50.397461+0000", 2024-04-19T02:03:54.871 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:54.871 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.871 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.871 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:54.871 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:50.397464+0000", 2024-04-19T02:03:54.871 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0000000000000001e-06 2024-04-19T02:03:54.871 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.871 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.871 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:54.872 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:50.399144+0000", 2024-04-19T02:03:54.872 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0016796269999999999 2024-04-19T02:03:54.872 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.872 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.872 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:54.872 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:50.399179+0000", 2024-04-19T02:03:54.872 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5265000000000002e-05 2024-04-19T02:03:54.872 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.872 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.872 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:54.872 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:50.399202+0000", 2024-04-19T02:03:54.872 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2775000000000001e-05 2024-04-19T02:03:54.872 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.872 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.872 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:54.873 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:50.399399+0000", 2024-04-19T02:03:54.873 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00019729599999999999 2024-04-19T02:03:54.873 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.873 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.873 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:54.873 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:50.399660+0000", 2024-04-19T02:03:54.873 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00026090299999999999 2024-04-19T02:03:54.873 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.873 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.873 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:54.873 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:50.403163+0000", 2024-04-19T02:03:54.873 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0035023139999999999 2024-04-19T02:03:54.873 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.873 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.873 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:54.873 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:50.409066+0000", 2024-04-19T02:03:54.874 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0059036399999999999 2024-04-19T02:03:54.874 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.874 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.874 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:54.874 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:50.414467+0000", 2024-04-19T02:03:54.874 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.005401164 2024-04-19T02:03:54.874 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:54.874 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:54.874 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:54.874 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:50.414564+0000", 2024-04-19T02:03:54.874 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6917999999999996e-05 2024-04-19T02:03:54.874 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.874 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.874 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.874 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:54.875 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:54.875 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:54.876 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-19T02:03:54.877 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:54.877+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:54.977 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:54.977+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:55.044 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:55.044 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:03:55.044 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-04-19T02:03:55.044 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:55.057 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-19T02:03:55.078 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:55.077+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:55.086 INFO:tasks.rados.rados.0.smithi059.stdout:306: done (2 left) 2024-04-19T02:03:55.086 INFO:tasks.rados.rados.0.smithi059.stdout:307: done (1 left) 2024-04-19T02:03:55.086 INFO:tasks.rados.rados.0.smithi059.stdout:308: done (0 left) 2024-04-19T02:03:55.086 INFO:tasks.rados.rados.0.smithi059.stdout:309: snap_remove snap 24 2024-04-19T02:03:55.179 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:55.181+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:55.225 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:55.226 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:03:55.226 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-04-19T02:03:55.226 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-04-19T02:03:55.226 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:55.241 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-19T02:03:55.280 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:55.281+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:55.365 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:55.365 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-04-19T02:03:55.365 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-04-19T02:03:55.365 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-04-19T02:03:55.365 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.365 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.4s2 42/26 ECSubWrite(tid=44, reqid=osd.1.0:44, at_version=42'32, trim_to=0'0, roll_forward_to=40'28))", 2024-04-19T02:03:55.365 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.739493+0000", 2024-04-19T02:03:55.365 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 28.62710951, 2024-04-19T02:03:55.365 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015165089, 2024-04-19T02:03:55.366 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.366 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.366 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:55.366 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.366 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.366 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.366 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.739493+0000", 2024-04-19T02:03:55.366 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.366 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.366 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.366 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.366 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.739493+0000", 2024-04-19T02:03:55.366 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.367 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.367 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.367 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.367 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.739495+0000", 2024-04-19T02:03:55.367 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8729999999999999e-06 2024-04-19T02:03:55.367 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.367 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.367 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.367 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.739550+0000", 2024-04-19T02:03:55.367 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5080000000000001e-05 2024-04-19T02:03:55.367 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.367 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.367 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.367 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.739584+0000", 2024-04-19T02:03:55.367 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3396999999999999e-05 2024-04-19T02:03:55.368 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.368 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.368 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.368 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.739611+0000", 2024-04-19T02:03:55.368 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7733999999999999e-05 2024-04-19T02:03:55.368 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.368 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.368 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.368 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.740435+0000", 2024-04-19T02:03:55.368 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00082331600000000004 2024-04-19T02:03:55.368 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.368 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.368 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:55.368 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.740463+0000", 2024-04-19T02:03:55.369 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7991999999999999e-05 2024-04-19T02:03:55.369 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.369 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.369 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:55.369 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.754535+0000", 2024-04-19T02:03:55.369 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014072092 2024-04-19T02:03:55.369 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.369 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.369 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.369 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.754658+0000", 2024-04-19T02:03:55.369 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012360499999999999 2024-04-19T02:03:55.369 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.369 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.369 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.370 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.370 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.370 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.1s4 47/26 ECSubWrite(tid=29, reqid=osd.0.0:29, at_version=47'15, trim_to=0'0, roll_forward_to=46'13))", 2024-04-19T02:03:55.370 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.087131+0000", 2024-04-19T02:03:55.370 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 24.279471615999999, 2024-04-19T02:03:55.370 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.028478961000000001, 2024-04-19T02:03:55.370 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.370 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.370 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:55.370 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.370 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.370 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.370 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087131+0000", 2024-04-19T02:03:55.371 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.371 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.371 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.371 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.371 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087131+0000", 2024-04-19T02:03:55.371 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.371 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.371 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.371 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.371 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087132+0000", 2024-04-19T02:03:55.371 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4000000000000001e-07 2024-04-19T02:03:55.371 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.371 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.371 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.371 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087185+0000", 2024-04-19T02:03:55.372 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3016999999999997e-05 2024-04-19T02:03:55.372 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.372 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.372 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.372 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087232+0000", 2024-04-19T02:03:55.372 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6989e-05 2024-04-19T02:03:55.372 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.372 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.372 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.372 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087259+0000", 2024-04-19T02:03:55.372 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7199000000000002e-05 2024-04-19T02:03:55.372 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.372 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.372 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.373 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087431+0000", 2024-04-19T02:03:55.373 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017193199999999999 2024-04-19T02:03:55.373 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.373 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.373 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:55.373 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087457+0000", 2024-04-19T02:03:55.373 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6148000000000001e-05 2024-04-19T02:03:55.373 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.373 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.373 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:55.373 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.115506+0000", 2024-04-19T02:03:55.373 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.028048812999999999 2024-04-19T02:03:55.373 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.373 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.373 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.373 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.115610+0000", 2024-04-19T02:03:55.374 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010412300000000001 2024-04-19T02:03:55.374 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.374 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.374 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.374 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.374 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.374 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.1s4 47/26 ECSubWrite(tid=30, reqid=osd.0.0:30, at_version=47'17, trim_to=0'0, roll_forward_to=46'13))", 2024-04-19T02:03:55.374 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.087466+0000", 2024-04-19T02:03:55.374 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 24.279137229, 2024-04-19T02:03:55.374 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.030894781, 2024-04-19T02:03:55.374 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.374 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.374 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:55.374 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.375 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.375 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.375 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087466+0000", 2024-04-19T02:03:55.375 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.375 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.375 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.375 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.375 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087466+0000", 2024-04-19T02:03:55.375 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.375 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.375 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.375 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.375 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087466+0000", 2024-04-19T02:03:55.375 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0400000000000005e-07 2024-04-19T02:03:55.375 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.376 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.376 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.376 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087524+0000", 2024-04-19T02:03:55.376 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7632e-05 2024-04-19T02:03:55.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.376 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.376 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.376 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087550+0000", 2024-04-19T02:03:55.376 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5871999999999999e-05 2024-04-19T02:03:55.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.376 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.376 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.376 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087570+0000", 2024-04-19T02:03:55.376 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0531999999999999e-05 2024-04-19T02:03:55.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.377 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.377 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.377 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.091486+0000", 2024-04-19T02:03:55.377 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039151009999999998 2024-04-19T02:03:55.377 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.377 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.377 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:55.377 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.091519+0000", 2024-04-19T02:03:55.377 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3874999999999997e-05 2024-04-19T02:03:55.377 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.377 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.377 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:55.377 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.118273+0000", 2024-04-19T02:03:55.377 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.026753835 2024-04-19T02:03:55.377 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.378 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.378 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.378 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.118360+0000", 2024-04-19T02:03:55.378 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7029999999999999e-05 2024-04-19T02:03:55.378 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.378 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.378 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.378 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.378 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.378 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s2 47/26 ECSubWrite(tid=181, reqid=osd.3.0:181, at_version=47'87, trim_to=0'0, roll_forward_to=46'85))", 2024-04-19T02:03:55.378 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.088687+0000", 2024-04-19T02:03:55.378 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 24.277915844999999, 2024-04-19T02:03:55.378 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.027025486000000001, 2024-04-19T02:03:55.378 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.379 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.379 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:55.379 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.379 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.379 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.379 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088687+0000", 2024-04-19T02:03:55.379 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.379 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.379 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.379 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.379 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088687+0000", 2024-04-19T02:03:55.379 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.379 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.379 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.379 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.380 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088687+0000", 2024-04-19T02:03:55.380 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7999999999999996e-07 2024-04-19T02:03:55.380 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.380 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.380 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.380 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088786+0000", 2024-04-19T02:03:55.380 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8510000000000004e-05 2024-04-19T02:03:55.380 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.380 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.380 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.380 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088822+0000", 2024-04-19T02:03:55.380 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6284e-05 2024-04-19T02:03:55.380 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.380 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.381 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.381 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088846+0000", 2024-04-19T02:03:55.381 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4125999999999999e-05 2024-04-19T02:03:55.381 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.381 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.381 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.381 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089107+0000", 2024-04-19T02:03:55.381 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00026044799999999998 2024-04-19T02:03:55.381 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.381 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.381 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:55.381 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089134+0000", 2024-04-19T02:03:55.381 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7067999999999999e-05 2024-04-19T02:03:55.381 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.381 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.381 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:55.382 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.115620+0000", 2024-04-19T02:03:55.382 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.026486464000000001 2024-04-19T02:03:55.382 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.382 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.382 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.382 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.115712+0000", 2024-04-19T02:03:55.382 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2106000000000002e-05 2024-04-19T02:03:55.382 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.382 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.382 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.382 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.382 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.383 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.bs5 47/26 ECSubWrite(tid=180, reqid=osd.3.0:180, at_version=47'36, trim_to=0'0, roll_forward_to=46'34))", 2024-04-19T02:03:55.383 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.088907+0000", 2024-04-19T02:03:55.383 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 24.277696236000001, 2024-04-19T02:03:55.383 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.028441793999999999, 2024-04-19T02:03:55.383 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.383 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.383 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:55.383 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.383 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.383 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.383 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088907+0000", 2024-04-19T02:03:55.383 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.383 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.383 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.383 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.384 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088907+0000", 2024-04-19T02:03:55.384 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.384 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.384 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.384 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.384 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088908+0000", 2024-04-19T02:03:55.384 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0100000000000001e-06 2024-04-19T02:03:55.384 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.384 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.384 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.384 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088952+0000", 2024-04-19T02:03:55.384 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4613000000000001e-05 2024-04-19T02:03:55.384 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.384 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.385 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.385 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088975+0000", 2024-04-19T02:03:55.385 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2518999999999999e-05 2024-04-19T02:03:55.385 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.385 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.385 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.385 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088993+0000", 2024-04-19T02:03:55.385 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7855000000000001e-05 2024-04-19T02:03:55.385 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.385 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.385 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.385 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089130+0000", 2024-04-19T02:03:55.385 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013709099999999999 2024-04-19T02:03:55.385 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.386 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.386 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:55.386 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089155+0000", 2024-04-19T02:03:55.386 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4924999999999999e-05 2024-04-19T02:03:55.386 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.386 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.386 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:55.386 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.117263+0000", 2024-04-19T02:03:55.386 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.028107953000000001 2024-04-19T02:03:55.386 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.386 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.387 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.387 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.117348+0000", 2024-04-19T02:03:55.387 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5828000000000006e-05 2024-04-19T02:03:55.387 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.387 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.387 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.387 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.387 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.387 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s2 47/26 ECSubWrite(tid=182, reqid=osd.3.0:182, at_version=47'89, trim_to=0'0, roll_forward_to=46'85))", 2024-04-19T02:03:55.387 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.089036+0000", 2024-04-19T02:03:55.387 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 24.277567208000001, 2024-04-19T02:03:55.387 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.027692491, 2024-04-19T02:03:55.388 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.388 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.388 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:55.388 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.388 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.388 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.388 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089036+0000", 2024-04-19T02:03:55.388 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.388 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.388 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.388 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.388 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089036+0000", 2024-04-19T02:03:55.388 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.388 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.389 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.389 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.389 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089036+0000", 2024-04-19T02:03:55.389 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7599999999999996e-07 2024-04-19T02:03:55.389 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.389 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.389 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.389 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089097+0000", 2024-04-19T02:03:55.389 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0223000000000003e-05 2024-04-19T02:03:55.389 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.389 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.389 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.389 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089117+0000", 2024-04-19T02:03:55.389 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0724999999999999e-05 2024-04-19T02:03:55.390 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.390 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.390 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.390 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089134+0000", 2024-04-19T02:03:55.390 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7065e-05 2024-04-19T02:03:55.390 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.390 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.390 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.390 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.091684+0000", 2024-04-19T02:03:55.390 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0025499780000000001 2024-04-19T02:03:55.390 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.390 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.390 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:55.390 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.091723+0000", 2024-04-19T02:03:55.390 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8787e-05 2024-04-19T02:03:55.391 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.391 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.391 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:55.391 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.116611+0000", 2024-04-19T02:03:55.391 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.024888321000000001 2024-04-19T02:03:55.391 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.391 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.391 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.391 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.116728+0000", 2024-04-19T02:03:55.391 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000116616 2024-04-19T02:03:55.391 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.391 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.391 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.391 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.391 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.392 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.bs5 47/26 ECSubWrite(tid=183, reqid=osd.3.0:183, at_version=47'38, trim_to=0'0, roll_forward_to=46'34))", 2024-04-19T02:03:55.392 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.089162+0000", 2024-04-19T02:03:55.392 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 24.277440841000001, 2024-04-19T02:03:55.392 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.031704567000000003, 2024-04-19T02:03:55.392 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.392 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.392 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:55.392 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.392 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.392 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.392 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089162+0000", 2024-04-19T02:03:55.392 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.392 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.392 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.392 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.393 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089162+0000", 2024-04-19T02:03:55.393 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.393 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.393 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.393 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.393 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089163+0000", 2024-04-19T02:03:55.393 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8999999999999996e-07 2024-04-19T02:03:55.393 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.393 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.393 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.393 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089208+0000", 2024-04-19T02:03:55.393 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5670999999999997e-05 2024-04-19T02:03:55.394 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.394 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.394 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.394 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089231+0000", 2024-04-19T02:03:55.394 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2898999999999999e-05 2024-04-19T02:03:55.394 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.394 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.394 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.394 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089248+0000", 2024-04-19T02:03:55.394 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7170999999999999e-05 2024-04-19T02:03:55.394 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.394 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.394 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.394 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.091699+0000", 2024-04-19T02:03:55.395 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0024503010000000002 2024-04-19T02:03:55.395 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.395 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.395 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:55.395 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.091731+0000", 2024-04-19T02:03:55.395 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2301000000000001e-05 2024-04-19T02:03:55.395 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.395 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.395 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:55.395 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.120784+0000", 2024-04-19T02:03:55.395 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.029052923000000001 2024-04-19T02:03:55.395 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.395 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.395 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.395 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.120866+0000", 2024-04-19T02:03:55.396 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2610999999999999e-05 2024-04-19T02:03:55.396 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.396 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.396 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.396 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.396 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.396 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.8s7 47/26 ECSubWrite(tid=86, reqid=osd.4.0:86, at_version=47'44, trim_to=0'0, roll_forward_to=46'42))", 2024-04-19T02:03:55.396 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.092233+0000", 2024-04-19T02:03:55.396 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 24.274370224999998, 2024-04-19T02:03:55.396 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.026209364999999998, 2024-04-19T02:03:55.396 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.396 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.396 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:55.396 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.397 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.397 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.397 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092233+0000", 2024-04-19T02:03:55.397 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.397 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.397 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.397 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.397 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092233+0000", 2024-04-19T02:03:55.397 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.397 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.397 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.397 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.397 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092233+0000", 2024-04-19T02:03:55.397 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8699999999999995e-07 2024-04-19T02:03:55.397 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.398 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.398 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.398 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092282+0000", 2024-04-19T02:03:55.398 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8869999999999998e-05 2024-04-19T02:03:55.398 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.398 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.398 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.398 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092311+0000", 2024-04-19T02:03:55.398 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9201e-05 2024-04-19T02:03:55.398 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.398 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.398 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.399 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092342+0000", 2024-04-19T02:03:55.399 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1275000000000001e-05 2024-04-19T02:03:55.399 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.399 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.399 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.399 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092597+0000", 2024-04-19T02:03:55.399 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00025494899999999997 2024-04-19T02:03:55.399 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.399 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.399 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:55.399 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092622+0000", 2024-04-19T02:03:55.399 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4355e-05 2024-04-19T02:03:55.399 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.399 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.400 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:55.400 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.118397+0000", 2024-04-19T02:03:55.400 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.025775005 2024-04-19T02:03:55.400 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.400 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.400 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.400 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.118442+0000", 2024-04-19T02:03:55.400 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5123000000000001e-05 2024-04-19T02:03:55.400 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.400 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.400 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.400 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.400 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.401 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:582 3.3s0 3:c26f510a:::smithi05930959-46:head [append 0~671744] snapc 10=[10,f,c] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-19T02:03:55.401 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:40.208006+0000", 2024-04-19T02:03:55.401 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 15.158596631, 2024-04-19T02:03:55.401 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022938303, 2024-04-19T02:03:55.401 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.401 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.401 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:55.401 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:55.401 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:55.401 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:55.401 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 582 2024-04-19T02:03:55.401 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.401 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.401 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.401 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.401 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.208006+0000", 2024-04-19T02:03:55.402 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.402 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.402 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.402 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.402 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.208006+0000", 2024-04-19T02:03:55.402 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.402 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.402 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.402 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.402 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.208009+0000", 2024-04-19T02:03:55.402 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2930000000000001e-06 2024-04-19T02:03:55.402 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.402 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.402 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.402 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.209365+0000", 2024-04-19T02:03:55.403 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001355745 2024-04-19T02:03:55.403 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.403 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.403 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.403 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.209399+0000", 2024-04-19T02:03:55.403 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3921000000000003e-05 2024-04-19T02:03:55.403 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.403 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.403 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.403 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.209414+0000", 2024-04-19T02:03:55.403 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4609e-05 2024-04-19T02:03:55.403 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.403 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.403 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.404 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.209587+0000", 2024-04-19T02:03:55.404 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017337400000000001 2024-04-19T02:03:55.404 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.404 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.404 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:55.404 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.209916+0000", 2024-04-19T02:03:55.404 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00032879699999999998 2024-04-19T02:03:55.404 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.404 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.404 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:55.404 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.212846+0000", 2024-04-19T02:03:55.404 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002929942 2024-04-19T02:03:55.404 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.404 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.404 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:55.404 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.230190+0000", 2024-04-19T02:03:55.405 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017344259000000001 2024-04-19T02:03:55.405 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.405 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.405 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:55.405 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.230733+0000", 2024-04-19T02:03:55.405 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00054272499999999998 2024-04-19T02:03:55.405 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.405 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.405 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.405 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.230944+0000", 2024-04-19T02:03:55.405 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000211638 2024-04-19T02:03:55.405 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.405 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.405 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.405 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.405 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.406 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:583 3.3s0 3:c26f510a:::smithi05930959-46:head [append 0~753664] snapc 10=[10,f,c] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-19T02:03:55.406 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:40.213899+0000", 2024-04-19T02:03:55.406 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 15.152704251999999, 2024-04-19T02:03:55.406 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018638891000000001, 2024-04-19T02:03:55.406 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.406 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.406 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:55.406 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:55.406 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:55.406 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:55.406 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 583 2024-04-19T02:03:55.406 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.406 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.407 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.407 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.407 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.213899+0000", 2024-04-19T02:03:55.407 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.407 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.407 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.407 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.407 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.213899+0000", 2024-04-19T02:03:55.407 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.407 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.407 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.407 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.407 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.213902+0000", 2024-04-19T02:03:55.407 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.653e-06 2024-04-19T02:03:55.407 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.408 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.408 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.408 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.215557+0000", 2024-04-19T02:03:55.408 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0016552800000000001 2024-04-19T02:03:55.408 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.408 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.408 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.408 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.215599+0000", 2024-04-19T02:03:55.408 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1111e-05 2024-04-19T02:03:55.408 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.408 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.408 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.408 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.215623+0000", 2024-04-19T02:03:55.408 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4834999999999999e-05 2024-04-19T02:03:55.409 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.409 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.409 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.409 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.216512+0000", 2024-04-19T02:03:55.409 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00088845700000000005 2024-04-19T02:03:55.409 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.409 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.409 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:55.409 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.216771+0000", 2024-04-19T02:03:55.409 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00025874300000000003 2024-04-19T02:03:55.409 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.409 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.409 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:55.409 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.219870+0000", 2024-04-19T02:03:55.410 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0030994759999999999 2024-04-19T02:03:55.410 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.410 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.410 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:55.410 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.230345+0000", 2024-04-19T02:03:55.410 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010474778000000001 2024-04-19T02:03:55.410 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.410 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.410 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:55.410 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.232336+0000", 2024-04-19T02:03:55.410 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0019915110000000001 2024-04-19T02:03:55.410 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.410 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.410 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.410 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.232537+0000", 2024-04-19T02:03:55.411 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020104699999999999 2024-04-19T02:03:55.411 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.411 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.411 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.411 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.411 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.411 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:584 3.3s0 3:c26f510a:::smithi05930959-46:head [append 0~409600] snapc 10=[10,f,c] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-19T02:03:55.411 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:40.217468+0000", 2024-04-19T02:03:55.411 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 15.14913466, 2024-04-19T02:03:55.411 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016044577000000001, 2024-04-19T02:03:55.411 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.411 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.411 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:55.411 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:55.411 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:55.412 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:55.412 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 584 2024-04-19T02:03:55.412 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.412 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.412 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.412 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.412 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.217468+0000", 2024-04-19T02:03:55.412 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.412 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.412 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.412 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.412 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.217468+0000", 2024-04-19T02:03:55.412 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.412 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.412 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.413 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.413 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.217471+0000", 2024-04-19T02:03:55.413 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7360000000000001e-06 2024-04-19T02:03:55.413 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.413 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.413 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.413 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218367+0000", 2024-04-19T02:03:55.413 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00089619299999999999 2024-04-19T02:03:55.413 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.413 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.413 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.413 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218402+0000", 2024-04-19T02:03:55.413 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4708e-05 2024-04-19T02:03:55.413 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.414 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.414 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.414 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218440+0000", 2024-04-19T02:03:55.414 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8362000000000003e-05 2024-04-19T02:03:55.414 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.414 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.414 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.414 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.222804+0000", 2024-04-19T02:03:55.414 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.004363596 2024-04-19T02:03:55.414 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.414 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.414 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:55.414 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.223100+0000", 2024-04-19T02:03:55.414 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00029668099999999997 2024-04-19T02:03:55.414 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.415 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.415 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:55.415 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.225367+0000", 2024-04-19T02:03:55.415 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0022668010000000001 2024-04-19T02:03:55.415 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.415 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.415 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:55.415 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.230348+0000", 2024-04-19T02:03:55.415 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0049803299999999998 2024-04-19T02:03:55.415 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.415 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.415 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:55.415 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.233363+0000", 2024-04-19T02:03:55.415 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0030157410000000002 2024-04-19T02:03:55.415 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.415 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.416 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.416 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.233513+0000", 2024-04-19T02:03:55.416 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000149429 2024-04-19T02:03:55.416 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.416 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.416 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.416 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.416 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.416 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:585 3.3s0 3:c26f510a:::smithi05930959-46:head [setxattr _header (57) in=7b] snapc 10=[10,f,c] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-19T02:03:55.416 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:40.218475+0000", 2024-04-19T02:03:55.416 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 15.148127300000001, 2024-04-19T02:03:55.416 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017451903000000001, 2024-04-19T02:03:55.416 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.416 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.416 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:55.416 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:55.416 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:55.417 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:55.417 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 585 2024-04-19T02:03:55.417 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.417 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.417 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.417 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.417 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218475+0000", 2024-04-19T02:03:55.417 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.417 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.417 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.417 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.417 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218475+0000", 2024-04-19T02:03:55.417 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.417 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.417 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.417 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.418 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218477+0000", 2024-04-19T02:03:55.418 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.66e-06 2024-04-19T02:03:55.418 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.418 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.418 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.418 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218486+0000", 2024-04-19T02:03:55.418 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9530000000000005e-06 2024-04-19T02:03:55.418 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.418 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.418 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.418 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218508+0000", 2024-04-19T02:03:55.418 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2294999999999999e-05 2024-04-19T02:03:55.418 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.418 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.419 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.419 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218523+0000", 2024-04-19T02:03:55.419 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.488e-05 2024-04-19T02:03:55.419 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.419 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.419 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.419 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.227831+0000", 2024-04-19T02:03:55.419 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.009308205 2024-04-19T02:03:55.419 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.419 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.419 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:55.419 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.228114+0000", 2024-04-19T02:03:55.419 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028219900000000001 2024-04-19T02:03:55.419 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.420 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.420 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:55.420 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.229580+0000", 2024-04-19T02:03:55.420 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001466799 2024-04-19T02:03:55.420 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.420 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.420 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:55.420 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.235272+0000", 2024-04-19T02:03:55.420 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0056910570000000002 2024-04-19T02:03:55.420 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.420 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.420 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:55.420 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.235624+0000", 2024-04-19T02:03:55.420 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00035226799999999998 2024-04-19T02:03:55.420 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.420 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.421 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.421 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.235927+0000", 2024-04-19T02:03:55.421 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00030358700000000002 2024-04-19T02:03:55.421 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.421 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.421 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.421 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.421 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.421 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:586 3.3s0 3:c26f510a:::smithi05930959-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56)", 2024-04-19T02:03:55.421 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:40.218545+0000", 2024-04-19T02:03:55.421 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 15.14805818, 2024-04-19T02:03:55.421 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.023206135999999999, 2024-04-19T02:03:55.421 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.421 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.421 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:55.421 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:55.422 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:55.422 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:55.422 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 586 2024-04-19T02:03:55.422 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.422 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.422 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.422 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.422 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218545+0000", 2024-04-19T02:03:55.422 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.422 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.422 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.422 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.422 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218545+0000", 2024-04-19T02:03:55.422 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.422 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.423 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.423 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.423 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218546+0000", 2024-04-19T02:03:55.423 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4160000000000001e-06 2024-04-19T02:03:55.423 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.423 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.423 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.423 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218553+0000", 2024-04-19T02:03:55.423 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1770000000000001e-06 2024-04-19T02:03:55.423 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.423 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.423 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.423 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218577+0000", 2024-04-19T02:03:55.423 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3635e-05 2024-04-19T02:03:55.423 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.424 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.424 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.424 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218593+0000", 2024-04-19T02:03:55.424 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5892999999999998e-05 2024-04-19T02:03:55.424 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.424 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.424 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.424 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.234991+0000", 2024-04-19T02:03:55.424 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016397954999999999 2024-04-19T02:03:55.424 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.424 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.424 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:55.424 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.235250+0000", 2024-04-19T02:03:55.424 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000259547 2024-04-19T02:03:55.425 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.425 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.425 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.425 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.236766+0000", 2024-04-19T02:03:55.425 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0015157599999999999 2024-04-19T02:03:55.425 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.425 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.425 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:55.425 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.237008+0000", 2024-04-19T02:03:55.425 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00024161300000000001 2024-04-19T02:03:55.425 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.425 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.425 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.425 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.241751+0000", 2024-04-19T02:03:55.425 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0047431399999999999 2024-04-19T02:03:55.426 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.426 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.426 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.426 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.426 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.426 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:765 3.5s0 3:a4ae10cb:::smithi05930959-29:head [append 0~770048] snapc 17=[17,16,15,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-19T02:03:55.426 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:51.089515+0000", 2024-04-19T02:03:55.426 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4.277088172, 2024-04-19T02:03:55.426 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.039942798000000002, 2024-04-19T02:03:55.426 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.426 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.426 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:55.426 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:55.426 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:55.427 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:55.427 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 765 2024-04-19T02:03:55.427 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.427 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.427 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.427 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.427 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.089515+0000", 2024-04-19T02:03:55.427 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.427 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.427 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.427 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.427 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.089515+0000", 2024-04-19T02:03:55.427 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.427 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.427 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.428 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.428 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.089518+0000", 2024-04-19T02:03:55.428 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.534e-06 2024-04-19T02:03:55.428 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.428 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.428 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.428 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.090847+0000", 2024-04-19T02:03:55.428 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0013285630000000001 2024-04-19T02:03:55.428 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.428 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.428 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.428 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.090883+0000", 2024-04-19T02:03:55.428 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6313000000000003e-05 2024-04-19T02:03:55.428 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.429 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.429 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.429 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.090908+0000", 2024-04-19T02:03:55.429 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5471999999999999e-05 2024-04-19T02:03:55.429 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.429 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.429 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.429 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.091083+0000", 2024-04-19T02:03:55.429 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000174403 2024-04-19T02:03:55.429 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.429 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.429 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op must wait for map", 2024-04-19T02:03:55.429 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.091096+0000", 2024-04-19T02:03:55.429 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3032e-05 2024-04-19T02:03:55.430 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.430 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.430 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.430 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.094803+0000", 2024-04-19T02:03:55.430 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0037069170000000001 2024-04-19T02:03:55.430 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.430 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.430 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:55.430 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.094961+0000", 2024-04-19T02:03:55.430 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015769699999999999 2024-04-19T02:03:55.430 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.430 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.430 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:55.430 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.097867+0000", 2024-04-19T02:03:55.431 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0029062480000000002 2024-04-19T02:03:55.431 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.431 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.431 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:55.431 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.116766+0000", 2024-04-19T02:03:55.431 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018899112999999999 2024-04-19T02:03:55.431 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.431 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.431 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:55.431 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.129316+0000", 2024-04-19T02:03:55.431 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012550318 2024-04-19T02:03:55.431 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.431 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.431 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.432 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.129457+0000", 2024-04-19T02:03:55.432 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014118799999999999 2024-04-19T02:03:55.432 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.432 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.432 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.432 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.432 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.432 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:766 3.5s0 3:a4ae10cb:::smithi05930959-29:head [append 0~540672] snapc 17=[17,16,15,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-19T02:03:55.432 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:51.093545+0000", 2024-04-19T02:03:55.432 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4.2730573879999998, 2024-04-19T02:03:55.432 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.040408039, 2024-04-19T02:03:55.432 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.432 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.432 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:55.432 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:55.433 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:55.433 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:55.433 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 766 2024-04-19T02:03:55.433 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.433 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.433 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.433 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.433 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.093545+0000", 2024-04-19T02:03:55.433 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.433 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.433 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.433 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.433 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.093545+0000", 2024-04-19T02:03:55.433 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.433 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.434 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.434 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.434 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.093548+0000", 2024-04-19T02:03:55.434 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.503e-06 2024-04-19T02:03:55.434 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.434 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.434 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.434 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.095040+0000", 2024-04-19T02:03:55.434 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0014918609999999999 2024-04-19T02:03:55.434 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.434 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.434 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.434 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.095062+0000", 2024-04-19T02:03:55.434 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1916000000000002e-05 2024-04-19T02:03:55.434 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.435 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.435 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.435 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.095076+0000", 2024-04-19T02:03:55.435 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4664999999999999e-05 2024-04-19T02:03:55.435 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.435 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.435 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.435 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.101526+0000", 2024-04-19T02:03:55.435 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0064491979999999997 2024-04-19T02:03:55.435 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.435 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.435 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:55.435 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.101864+0000", 2024-04-19T02:03:55.435 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00033830999999999998 2024-04-19T02:03:55.435 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.436 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.436 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:55.436 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.104979+0000", 2024-04-19T02:03:55.436 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0031149459999999999 2024-04-19T02:03:55.436 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.436 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.436 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:55.436 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.116769+0000", 2024-04-19T02:03:55.436 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011790512 2024-04-19T02:03:55.436 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.436 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.436 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:55.436 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.133770+0000", 2024-04-19T02:03:55.436 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017000347999999998 2024-04-19T02:03:55.436 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.437 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.437 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.437 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.133953+0000", 2024-04-19T02:03:55.437 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00018378000000000001 2024-04-19T02:03:55.437 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.437 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.437 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.437 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.437 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.437 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:767 3.5s0 3:a4ae10cb:::smithi05930959-29:head [append 0~753664] snapc 17=[17,16,15,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-19T02:03:55.437 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:51.099323+0000", 2024-04-19T02:03:55.437 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4.2672801939999996, 2024-04-19T02:03:55.437 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.035037549000000001, 2024-04-19T02:03:55.437 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.437 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.438 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:55.438 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:55.438 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:55.438 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:55.438 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 767 2024-04-19T02:03:55.438 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.438 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.438 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.438 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.438 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.099323+0000", 2024-04-19T02:03:55.438 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.438 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.438 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.438 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.438 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.099323+0000", 2024-04-19T02:03:55.439 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.439 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.439 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.439 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.439 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.099333+0000", 2024-04-19T02:03:55.439 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0412e-05 2024-04-19T02:03:55.439 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.439 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.439 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.439 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.100798+0000", 2024-04-19T02:03:55.439 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001465166 2024-04-19T02:03:55.440 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.440 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.440 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.440 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.100841+0000", 2024-04-19T02:03:55.440 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2837999999999999e-05 2024-04-19T02:03:55.440 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.440 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.440 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.440 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.100865+0000", 2024-04-19T02:03:55.440 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4241999999999999e-05 2024-04-19T02:03:55.440 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.440 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.440 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.440 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.107634+0000", 2024-04-19T02:03:55.441 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0067688389999999996 2024-04-19T02:03:55.441 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.441 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.441 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:55.441 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.107945+0000", 2024-04-19T02:03:55.441 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00031116999999999999 2024-04-19T02:03:55.441 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.441 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.441 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:55.441 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.114382+0000", 2024-04-19T02:03:55.441 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0064363579999999997 2024-04-19T02:03:55.441 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.441 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.441 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:55.442 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.118042+0000", 2024-04-19T02:03:55.442 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0036606849999999999 2024-04-19T02:03:55.442 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.442 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.442 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:55.442 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.134197+0000", 2024-04-19T02:03:55.442 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016155124999999999 2024-04-19T02:03:55.442 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.442 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.442 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.442 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.134360+0000", 2024-04-19T02:03:55.442 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000162714 2024-04-19T02:03:55.442 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.443 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.443 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.443 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.443 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.443 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:768 3.5s0 3:a4ae10cb:::smithi05930959-29:head [append 0~262144] snapc 17=[17,16,15,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-19T02:03:55.443 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:51.101861+0000", 2024-04-19T02:03:55.443 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4.2647416849999997, 2024-04-19T02:03:55.443 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.032826545999999998, 2024-04-19T02:03:55.443 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.443 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.443 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:55.443 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:55.443 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:55.443 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:55.443 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 768 2024-04-19T02:03:55.443 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.444 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.444 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.444 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.444 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.101861+0000", 2024-04-19T02:03:55.444 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.444 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.444 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.444 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.444 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.101861+0000", 2024-04-19T02:03:55.444 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.444 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.444 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.444 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.444 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.101866+0000", 2024-04-19T02:03:55.444 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4399999999999998e-06 2024-04-19T02:03:55.444 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.444 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.444 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.445 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102379+0000", 2024-04-19T02:03:55.445 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00051389600000000001 2024-04-19T02:03:55.445 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.445 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.445 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.445 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102412+0000", 2024-04-19T02:03:55.445 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2832e-05 2024-04-19T02:03:55.445 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.445 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.445 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.445 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102439+0000", 2024-04-19T02:03:55.445 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6842000000000002e-05 2024-04-19T02:03:55.445 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.445 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.446 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.446 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.118803+0000", 2024-04-19T02:03:55.446 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016363637 2024-04-19T02:03:55.446 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.446 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.446 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:55.446 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.119080+0000", 2024-04-19T02:03:55.446 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00027685600000000002 2024-04-19T02:03:55.446 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.446 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.446 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:55.446 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.121701+0000", 2024-04-19T02:03:55.446 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0026218389999999999 2024-04-19T02:03:55.446 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.446 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.447 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:55.447 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.127449+0000", 2024-04-19T02:03:55.447 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.005747149 2024-04-19T02:03:55.447 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.447 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.447 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:55.447 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.134572+0000", 2024-04-19T02:03:55.447 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0071231269999999999 2024-04-19T02:03:55.447 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.447 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.447 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.447 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.134688+0000", 2024-04-19T02:03:55.447 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011592800000000001 2024-04-19T02:03:55.447 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.447 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.447 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.448 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.448 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.448 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:769 3.5s0 3:a4ae10cb:::smithi05930959-29:head [setxattr _header (57) in=7b] snapc 17=[17,16,15,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-19T02:03:55.448 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:51.102479+0000", 2024-04-19T02:03:55.448 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4.2641237329999999, 2024-04-19T02:03:55.448 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.032545561000000001, 2024-04-19T02:03:55.448 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.448 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.448 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:55.448 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:55.448 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:55.448 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:55.448 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 769 2024-04-19T02:03:55.448 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.449 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.449 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.449 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.449 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102479+0000", 2024-04-19T02:03:55.449 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.449 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.449 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.449 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.449 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102479+0000", 2024-04-19T02:03:55.449 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.449 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.449 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.449 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.449 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102481+0000", 2024-04-19T02:03:55.450 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.843e-06 2024-04-19T02:03:55.450 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.450 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.450 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.450 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102492+0000", 2024-04-19T02:03:55.450 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0793e-05 2024-04-19T02:03:55.450 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.450 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.450 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.450 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102516+0000", 2024-04-19T02:03:55.450 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3873e-05 2024-04-19T02:03:55.450 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.450 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.450 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.451 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102532+0000", 2024-04-19T02:03:55.451 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6674999999999999e-05 2024-04-19T02:03:55.451 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.451 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.451 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.451 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.123043+0000", 2024-04-19T02:03:55.451 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020510626000000001 2024-04-19T02:03:55.451 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.451 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.451 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:55.451 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.123345+0000", 2024-04-19T02:03:55.451 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000302202 2024-04-19T02:03:55.451 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.451 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.451 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:55.451 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.124785+0000", 2024-04-19T02:03:55.452 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001439596 2024-04-19T02:03:55.452 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.452 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.452 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:55.452 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.127574+0000", 2024-04-19T02:03:55.452 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002789822 2024-04-19T02:03:55.452 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.452 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.452 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:55.452 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.134878+0000", 2024-04-19T02:03:55.452 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0073030400000000002 2024-04-19T02:03:55.452 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.452 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.453 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.453 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.135025+0000", 2024-04-19T02:03:55.453 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014709099999999999 2024-04-19T02:03:55.453 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.453 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.453 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.453 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.453 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.453 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:770 3.5s0 3:a4ae10cb:::smithi05930959-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e67)", 2024-04-19T02:03:55.453 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:51.102554+0000", 2024-04-19T02:03:55.453 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4.2640485799999999, 2024-04-19T02:03:55.453 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.037176628000000003, 2024-04-19T02:03:55.453 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.453 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.453 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:55.454 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:55.454 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:55.454 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:55.454 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 770 2024-04-19T02:03:55.454 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.454 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.454 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.454 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.454 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102554+0000", 2024-04-19T02:03:55.454 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.454 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.454 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.454 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.454 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102554+0000", 2024-04-19T02:03:55.455 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.455 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.455 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.455 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.455 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102555+0000", 2024-04-19T02:03:55.455 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1799999999999999e-06 2024-04-19T02:03:55.455 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.455 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.455 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.455 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102562+0000", 2024-04-19T02:03:55.455 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5039999999999999e-06 2024-04-19T02:03:55.455 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.455 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.455 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.455 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102585+0000", 2024-04-19T02:03:55.456 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3159e-05 2024-04-19T02:03:55.456 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.456 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.456 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.456 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102602+0000", 2024-04-19T02:03:55.456 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6892e-05 2024-04-19T02:03:55.456 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.456 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.456 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.456 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.127057+0000", 2024-04-19T02:03:55.456 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0244551 2024-04-19T02:03:55.456 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.456 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.456 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:55.456 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.127389+0000", 2024-04-19T02:03:55.457 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00033189999999999999 2024-04-19T02:03:55.457 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.457 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.457 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.457 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.136002+0000", 2024-04-19T02:03:55.457 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0086132810000000004 2024-04-19T02:03:55.457 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.457 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.457 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:55.457 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.136156+0000", 2024-04-19T02:03:55.457 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015367399999999999 2024-04-19T02:03:55.457 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.457 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.457 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.457 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.139731+0000", 2024-04-19T02:03:55.458 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0035749380000000002 2024-04-19T02:03:55.458 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.458 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.458 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.458 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.458 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.458 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 67/26 ECSubWriteReply(tid=85, last_complete=67'41, committed=1, applied=1))", 2024-04-19T02:03:55.458 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:51.103036+0000", 2024-04-19T02:03:55.458 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4.2635669749999998, 2024-04-19T02:03:55.458 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013736781, 2024-04-19T02:03:55.458 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.458 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.458 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:55.458 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.458 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.459 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.459 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.103036+0000", 2024-04-19T02:03:55.459 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.459 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.459 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.459 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.459 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.103036+0000", 2024-04-19T02:03:55.459 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.459 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.459 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.459 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.459 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.103038+0000", 2024-04-19T02:03:55.459 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1500000000000002e-06 2024-04-19T02:03:55.459 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.460 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.460 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.460 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.103050+0000", 2024-04-19T02:03:55.460 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2046e-05 2024-04-19T02:03:55.460 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.460 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.460 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.460 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.103071+0000", 2024-04-19T02:03:55.460 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1373e-05 2024-04-19T02:03:55.460 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.460 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.460 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.460 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.103085+0000", 2024-04-19T02:03:55.460 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3199e-05 2024-04-19T02:03:55.460 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.460 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.461 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.461 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.116714+0000", 2024-04-19T02:03:55.461 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013629245 2024-04-19T02:03:55.461 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.461 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.461 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.461 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.116773+0000", 2024-04-19T02:03:55.461 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8767999999999997e-05 2024-04-19T02:03:55.461 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.461 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.461 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.461 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.461 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.461 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:55.463 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-19T02:03:55.463 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:55.465+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:55.563 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:55.565+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:55.632 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:55.632 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:03:55.632 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-04-19T02:03:55.632 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:55.646 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-19T02:03:55.690 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:55.693+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:55.763 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:55.764 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:03:55.764 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-04-19T02:03:55.764 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-04-19T02:03:55.764 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:55.764 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:55.765+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:55.777 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-19T02:03:55.865 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:55.865+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:55.926 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T02:03:55.947 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:03:55.947 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-04-19T02:03:55.947 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-04-19T02:03:55.947 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-04-19T02:03:55.947 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.947 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/26 ECSubWrite(tid=132, reqid=osd.3.0:132, at_version=42'63, trim_to=0'0, roll_forward_to=40'61))", 2024-04-19T02:03:55.947 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.748602+0000", 2024-04-19T02:03:55.947 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 29.198731859999999, 2024-04-19T02:03:55.947 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.024032777000000002, 2024-04-19T02:03:55.947 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.947 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.948 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:55.948 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.948 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.948 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.948 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748602+0000", 2024-04-19T02:03:55.948 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.948 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.948 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.948 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.948 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748602+0000", 2024-04-19T02:03:55.948 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.948 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.948 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.948 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.948 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748603+0000", 2024-04-19T02:03:55.949 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4600000000000003e-07 2024-04-19T02:03:55.949 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.949 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.949 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.949 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748657+0000", 2024-04-19T02:03:55.949 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3803e-05 2024-04-19T02:03:55.949 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.949 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.949 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.949 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748688+0000", 2024-04-19T02:03:55.949 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0871000000000003e-05 2024-04-19T02:03:55.949 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.949 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.949 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.950 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748710+0000", 2024-04-19T02:03:55.950 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2634999999999999e-05 2024-04-19T02:03:55.950 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.950 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.950 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.950 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748980+0000", 2024-04-19T02:03:55.950 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00026984900000000001 2024-04-19T02:03:55.950 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.950 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.950 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:55.950 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.749007+0000", 2024-04-19T02:03:55.950 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6225e-05 2024-04-19T02:03:55.950 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.950 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.951 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:55.951 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.772533+0000", 2024-04-19T02:03:55.951 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.023526782 2024-04-19T02:03:55.951 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.951 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.951 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.951 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.772635+0000", 2024-04-19T02:03:55.951 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000101766 2024-04-19T02:03:55.951 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.951 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.951 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.951 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.951 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.951 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.8s3 42/26 ECSubWrite(tid=71, reqid=osd.4.0:71, at_version=42'33, trim_to=0'0, roll_forward_to=37'31))", 2024-04-19T02:03:55.952 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.751483+0000", 2024-04-19T02:03:55.952 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 29.195851407999999, 2024-04-19T02:03:55.952 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.024697106999999999, 2024-04-19T02:03:55.952 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.952 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.952 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:55.952 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.952 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.952 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.952 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751483+0000", 2024-04-19T02:03:55.952 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.952 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.952 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.952 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.952 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751483+0000", 2024-04-19T02:03:55.953 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.953 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.953 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.953 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.953 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751483+0000", 2024-04-19T02:03:55.953 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9999999999999997e-07 2024-04-19T02:03:55.953 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.953 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.953 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.953 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751545+0000", 2024-04-19T02:03:55.953 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1958999999999994e-05 2024-04-19T02:03:55.953 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.953 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.953 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.954 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755035+0000", 2024-04-19T02:03:55.954 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0034897819999999999 2024-04-19T02:03:55.954 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.954 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.954 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.954 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755064+0000", 2024-04-19T02:03:55.954 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9227999999999999e-05 2024-04-19T02:03:55.954 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.954 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.954 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.954 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755228+0000", 2024-04-19T02:03:55.954 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016367700000000001 2024-04-19T02:03:55.954 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.954 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.954 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:55.955 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755256+0000", 2024-04-19T02:03:55.955 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7900999999999999e-05 2024-04-19T02:03:55.955 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.955 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.955 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:55.955 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.776106+0000", 2024-04-19T02:03:55.955 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020850560000000001 2024-04-19T02:03:55.955 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.955 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.955 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.955 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.776180+0000", 2024-04-19T02:03:55.955 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3399999999999995e-05 2024-04-19T02:03:55.956 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.956 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.956 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.956 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.956 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.956 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/26 ECSubWrite(tid=134, reqid=osd.3.0:134, at_version=42'65, trim_to=0'0, roll_forward_to=40'61))", 2024-04-19T02:03:55.956 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.751576+0000", 2024-04-19T02:03:55.956 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 29.195757701000002, 2024-04-19T02:03:55.956 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.027269287, 2024-04-19T02:03:55.956 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.956 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.956 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:55.956 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.956 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.957 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.957 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751576+0000", 2024-04-19T02:03:55.957 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.957 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.957 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.957 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.957 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751576+0000", 2024-04-19T02:03:55.957 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.957 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.957 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.957 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.957 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751578+0000", 2024-04-19T02:03:55.957 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9659999999999999e-06 2024-04-19T02:03:55.957 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.958 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.958 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.958 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751651+0000", 2024-04-19T02:03:55.958 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2781000000000007e-05 2024-04-19T02:03:55.958 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.958 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.958 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.958 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755528+0000", 2024-04-19T02:03:55.958 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0038770359999999999 2024-04-19T02:03:55.958 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.958 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.958 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.958 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755558+0000", 2024-04-19T02:03:55.958 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9414e-05 2024-04-19T02:03:55.958 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.959 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.959 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.959 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755853+0000", 2024-04-19T02:03:55.959 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00029508699999999998 2024-04-19T02:03:55.959 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.959 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.959 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:55.959 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755881+0000", 2024-04-19T02:03:55.959 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7965000000000001e-05 2024-04-19T02:03:55.959 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.959 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.959 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:55.959 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778764+0000", 2024-04-19T02:03:55.960 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022883244000000001 2024-04-19T02:03:55.960 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.960 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.960 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.960 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778846+0000", 2024-04-19T02:03:55.960 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1793999999999996e-05 2024-04-19T02:03:55.960 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.960 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.960 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.960 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.960 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.960 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:371 3.as0 3:54128c7a:::smithi05930959-20:head [read 0~1605632,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:55.960 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.760138+0000", 2024-04-19T02:03:55.961 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 29.187196270000001, 2024-04-19T02:03:55.961 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.031633367000000003, 2024-04-19T02:03:55.961 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.961 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.961 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:55.961 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:55.961 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:55.961 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:55.961 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 371 2024-04-19T02:03:55.961 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.961 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.961 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.961 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.961 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760138+0000", 2024-04-19T02:03:55.961 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.962 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.962 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.962 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.962 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760138+0000", 2024-04-19T02:03:55.962 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.962 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.962 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.962 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.962 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760141+0000", 2024-04-19T02:03:55.962 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0790000000000001e-06 2024-04-19T02:03:55.962 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.962 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.962 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.962 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760155+0000", 2024-04-19T02:03:55.963 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4083000000000001e-05 2024-04-19T02:03:55.963 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.963 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.963 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.963 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760187+0000", 2024-04-19T02:03:55.963 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1964000000000001e-05 2024-04-19T02:03:55.963 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.963 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.963 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.963 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760208+0000", 2024-04-19T02:03:55.963 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1395000000000001e-05 2024-04-19T02:03:55.963 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.963 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.963 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.963 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760524+0000", 2024-04-19T02:03:55.964 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00031564199999999998 2024-04-19T02:03:55.964 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.964 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.964 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:55.964 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760793+0000", 2024-04-19T02:03:55.964 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000268819 2024-04-19T02:03:55.964 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.964 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.964 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.964 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774492+0000", 2024-04-19T02:03:55.964 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013699065999999999 2024-04-19T02:03:55.964 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.964 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.964 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:55.964 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774730+0000", 2024-04-19T02:03:55.964 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00023788499999999999 2024-04-19T02:03:55.964 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.965 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.965 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.965 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.791771+0000", 2024-04-19T02:03:55.965 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017041434000000001 2024-04-19T02:03:55.965 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.965 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.965 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.965 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.965 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.965 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:372 3.as0 3:54128c7a:::smithi05930959-20:head [read 0~1605632,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:55.965 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.760260+0000", 2024-04-19T02:03:55.965 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 29.187073903000002, 2024-04-19T02:03:55.965 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.034139714000000002, 2024-04-19T02:03:55.965 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.965 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.965 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:55.965 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:55.966 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:55.966 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:55.966 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 372 2024-04-19T02:03:55.966 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.966 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.966 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.966 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.966 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760260+0000", 2024-04-19T02:03:55.966 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.966 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.966 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.966 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.966 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760260+0000", 2024-04-19T02:03:55.966 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.967 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.967 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.967 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.967 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760262+0000", 2024-04-19T02:03:55.967 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6029999999999999e-06 2024-04-19T02:03:55.967 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.967 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.967 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.967 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760272+0000", 2024-04-19T02:03:55.967 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6560000000000004e-06 2024-04-19T02:03:55.967 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.967 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.967 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.967 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760298+0000", 2024-04-19T02:03:55.967 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6625e-05 2024-04-19T02:03:55.968 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.968 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.968 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.968 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760316+0000", 2024-04-19T02:03:55.968 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7459e-05 2024-04-19T02:03:55.968 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.968 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.968 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.968 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760885+0000", 2024-04-19T02:03:55.968 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00056941200000000004 2024-04-19T02:03:55.968 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.968 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.968 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:55.968 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761126+0000", 2024-04-19T02:03:55.969 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000241124 2024-04-19T02:03:55.969 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.969 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.969 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.969 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.775702+0000", 2024-04-19T02:03:55.969 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014575764 2024-04-19T02:03:55.969 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.969 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.969 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:55.969 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.777674+0000", 2024-04-19T02:03:55.969 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001972297 2024-04-19T02:03:55.969 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.969 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.969 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.970 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.794400+0000", 2024-04-19T02:03:55.970 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016725773999999999 2024-04-19T02:03:55.970 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.970 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.970 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.970 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.970 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.970 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:373 3.as0 3:54128c7a:::smithi05930959-20:head [read 0~1605632,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:03:55.970 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.760341+0000", 2024-04-19T02:03:55.970 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 29.186993448999999, 2024-04-19T02:03:55.970 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.043665984999999997, 2024-04-19T02:03:55.970 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.971 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.971 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:55.971 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:55.971 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:55.971 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:55.971 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 373 2024-04-19T02:03:55.971 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.971 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.971 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.971 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.971 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760341+0000", 2024-04-19T02:03:55.971 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.971 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.971 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.972 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.972 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760341+0000", 2024-04-19T02:03:55.972 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.972 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.972 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.972 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.972 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760342+0000", 2024-04-19T02:03:55.972 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.497e-06 2024-04-19T02:03:55.972 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.972 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.972 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.972 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760350+0000", 2024-04-19T02:03:55.972 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1119999999999996e-06 2024-04-19T02:03:55.972 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.972 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.973 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.973 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760370+0000", 2024-04-19T02:03:55.973 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9219e-05 2024-04-19T02:03:55.973 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.973 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.973 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.973 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760385+0000", 2024-04-19T02:03:55.973 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5905000000000001e-05 2024-04-19T02:03:55.973 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.973 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.973 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.973 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761214+0000", 2024-04-19T02:03:55.973 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000828939 2024-04-19T02:03:55.973 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.974 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.974 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:55.974 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761439+0000", 2024-04-19T02:03:55.974 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022493900000000001 2024-04-19T02:03:55.974 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.974 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.974 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.974 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.794533+0000", 2024-04-19T02:03:55.974 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.033094077999999999 2024-04-19T02:03:55.974 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.974 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.974 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:55.975 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.794696+0000", 2024-04-19T02:03:55.975 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000162442 2024-04-19T02:03:55.975 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.975 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.975 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.975 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.804007+0000", 2024-04-19T02:03:55.975 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0093108540000000004 2024-04-19T02:03:55.975 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.975 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.975 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.975 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.975 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.975 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:381 3.as0 3:512144ec:::smithi05930959-48:head [create,setxattr pnc]aaY (11788) in=7b,setxattr ZTVVStUpk__YaS\\blR\\XgesYT[Or[[aUett_nil] (1859) in=40b,setxattr ZtotWOVcPP_oiTUUTlqtYZsNneQnmhc\\bNdNpsgo (1402) in=40b,setxattr cR\\akUitgYfao[oisZ^ach\\i_kdge\\[c_mbmUf]^ (313) in=40b,setxattr pRe[UcWhXR^oagaSYScYPQa]r[hdldO\\S`V^[nh^ (874) in=40b,setxattr srVXOoW^mbi\\ccUpc`barbsc (11788) in=24b,setxattr pqjhUkafUjRqdpObgi_ZX^QV (11788) in=24b,setxattr q]Of\\bd_VgdoaP\\k^tmmpUqq^otrVVOdrtgU]nVk (12280) in=40b,setxattr ]Ple__\\nrXPQYecmQXf_o^bgN`mofssSPQrYslSO (1859) in=40b,setxattr ZiQcjT`UZQ]es__dmo^h]NdV^`qT_hniQtVY`WsT (770) in=40b,setxattr gpT^oWmeYZpbVdojjWnUWO[YfZptiQhdWdhs]Zo^ (841) in=40b,setxattr `Zgeqf\\Rb][aUdTVXYhj[ealjcUnRaiY (11788) in=32b,setxattr [dN]aooYVPYS^Q`_`bqSNVSS^`ZT\\csYZSrnjtUT (11296) in=40b,setxattr ehcNktRfRjqWWi[^XWcZokN]Wb[YnkeNl\\lkt_t\\ (1262) in=40b,setxattr OdsZcnit[de]qNV (11788) in=15b,setxattr ^]^lenZo\\TjoZOpONRno]jtaWOd`tqNdWcYccWNk (560) in=40b,setxattr ZUohXUqoPmYaaTceWo\\SRigth\\qP[Tg_^^asjknX (1085) in=40b,setxattr lR`dUarlc[n`ctj^TlY\\QpagN_doPfV`lZQ^kNtY (526) in=40b,setxattr [iXNTskqfmN\\qVhqrm`rdXpaN_VaeRbd^O\\Z_gcY (1261) in=40b,setxattr t_jR_akNV_aXVR^icaga]cP]baNne_f_UdsoeVUk (1894) in=40b] snapc 9=[8,6] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T02:03:55.976 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:27.087092+0000", 2024-04-19T02:03:55.976 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 28.860242455000002, 2024-04-19T02:03:55.976 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.025861967999999999, 2024-04-19T02:03:55.976 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.976 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.976 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:55.976 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:55.976 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:55.976 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:55.976 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 381 2024-04-19T02:03:55.976 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.976 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.976 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.976 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.977 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087092+0000", 2024-04-19T02:03:55.977 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.977 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.977 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.977 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.977 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087092+0000", 2024-04-19T02:03:55.977 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.977 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.977 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.977 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.977 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087095+0000", 2024-04-19T02:03:55.977 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2799999999999999e-06 2024-04-19T02:03:55.977 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.978 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.978 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.978 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087306+0000", 2024-04-19T02:03:55.978 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021136000000000001 2024-04-19T02:03:55.978 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.978 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.978 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.978 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087341+0000", 2024-04-19T02:03:55.978 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4654999999999999e-05 2024-04-19T02:03:55.978 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.978 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.978 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.978 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087358+0000", 2024-04-19T02:03:55.979 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7261e-05 2024-04-19T02:03:55.979 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.979 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.979 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.979 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087536+0000", 2024-04-19T02:03:55.979 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000177424 2024-04-19T02:03:55.979 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.979 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.979 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op must wait for map", 2024-04-19T02:03:55.979 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087551+0000", 2024-04-19T02:03:55.979 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5362e-05 2024-04-19T02:03:55.979 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.979 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.979 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.979 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.092762+0000", 2024-04-19T02:03:55.980 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0052113589999999996 2024-04-19T02:03:55.980 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.980 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.980 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:55.980 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.092952+0000", 2024-04-19T02:03:55.980 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00018992100000000001 2024-04-19T02:03:55.980 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.980 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.980 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:55.980 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.095255+0000", 2024-04-19T02:03:55.980 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0023027030000000001 2024-04-19T02:03:55.980 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.980 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.980 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:55.980 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.101758+0000", 2024-04-19T02:03:55.981 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0065034380000000003 2024-04-19T02:03:55.981 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.981 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.981 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:55.981 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.112669+0000", 2024-04-19T02:03:55.981 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010910737 2024-04-19T02:03:55.981 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.981 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.981 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.981 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.112954+0000", 2024-04-19T02:03:55.981 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028446800000000001 2024-04-19T02:03:55.981 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.982 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.982 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.982 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.982 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.982 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:434 3.as0 3:5a72bad2:::smithi05930959-41:head [read 0~3293184,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:55.982 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:30.194592+0000", 2024-04-19T02:03:55.982 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 25.752742496, 2024-04-19T02:03:55.982 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020994932000000001, 2024-04-19T02:03:55.982 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.982 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.982 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:55.982 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:55.982 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:55.982 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:55.983 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 434 2024-04-19T02:03:55.983 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.983 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.983 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.983 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.983 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194592+0000", 2024-04-19T02:03:55.983 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.983 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.983 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.983 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.983 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194592+0000", 2024-04-19T02:03:55.983 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.983 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.984 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.984 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.984 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194594+0000", 2024-04-19T02:03:55.984 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6730000000000001e-06 2024-04-19T02:03:55.984 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.984 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.984 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.984 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194608+0000", 2024-04-19T02:03:55.984 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3382e-05 2024-04-19T02:03:55.984 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.984 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.984 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.984 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194638+0000", 2024-04-19T02:03:55.984 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0691000000000001e-05 2024-04-19T02:03:55.984 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.985 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.985 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.985 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194659+0000", 2024-04-19T02:03:55.985 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0182000000000001e-05 2024-04-19T02:03:55.985 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.985 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.985 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.985 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194837+0000", 2024-04-19T02:03:55.985 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017831800000000001 2024-04-19T02:03:55.985 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.985 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.985 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:55.985 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194983+0000", 2024-04-19T02:03:55.985 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014610299999999999 2024-04-19T02:03:55.985 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.985 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.986 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.986 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.215587+0000", 2024-04-19T02:03:55.986 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020603583000000002 2024-04-19T02:03:55.986 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.986 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.986 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.986 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.986 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.986 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:435 3.as0 3:5a72bad2:::smithi05930959-41:head [read 0~3293184,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:55.986 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:30.194714+0000", 2024-04-19T02:03:55.986 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 25.752620615000001, 2024-04-19T02:03:55.986 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.025824666, 2024-04-19T02:03:55.986 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.987 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.987 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:55.987 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:55.987 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:55.987 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:55.987 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 435 2024-04-19T02:03:55.987 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.987 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.987 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.987 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.987 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194714+0000", 2024-04-19T02:03:55.987 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.987 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.987 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.987 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.987 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194714+0000", 2024-04-19T02:03:55.988 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.988 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.988 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.988 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.988 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194715+0000", 2024-04-19T02:03:55.988 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.497e-06 2024-04-19T02:03:55.988 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.988 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.988 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.988 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194727+0000", 2024-04-19T02:03:55.988 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2021999999999999e-05 2024-04-19T02:03:55.988 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.988 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.988 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.988 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194772+0000", 2024-04-19T02:03:55.989 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4620000000000003e-05 2024-04-19T02:03:55.989 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.989 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.989 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.989 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194790+0000", 2024-04-19T02:03:55.989 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7846000000000002e-05 2024-04-19T02:03:55.989 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.989 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.989 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.989 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.195633+0000", 2024-04-19T02:03:55.989 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00084348400000000003 2024-04-19T02:03:55.989 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.989 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.989 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:55.990 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.195885+0000", 2024-04-19T02:03:55.990 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00025215900000000001 2024-04-19T02:03:55.990 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.990 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.990 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.990 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.220538+0000", 2024-04-19T02:03:55.990 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.024653037999999999 2024-04-19T02:03:55.990 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.990 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.990 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.990 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.990 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.990 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:436 3.as0 3:5a72bad2:::smithi05930959-41:head [read 0~3293184,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:03:55.990 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:30.194819+0000", 2024-04-19T02:03:55.991 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 25.752515152000001, 2024-04-19T02:03:55.991 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.028970988999999999, 2024-04-19T02:03:55.991 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.991 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.991 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:55.991 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:55.991 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:55.991 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:55.991 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 436 2024-04-19T02:03:55.991 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.991 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.991 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.991 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.991 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194819+0000", 2024-04-19T02:03:55.992 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.992 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.992 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.992 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.992 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194819+0000", 2024-04-19T02:03:55.992 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.992 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.992 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.992 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.992 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194820+0000", 2024-04-19T02:03:55.992 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1829999999999999e-06 2024-04-19T02:03:55.992 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.992 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.992 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.993 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194828+0000", 2024-04-19T02:03:55.993 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.306e-06 2024-04-19T02:03:55.993 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.993 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.993 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.993 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194846+0000", 2024-04-19T02:03:55.993 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8592000000000001e-05 2024-04-19T02:03:55.993 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.993 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.993 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.993 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194860+0000", 2024-04-19T02:03:55.993 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4185999999999999e-05 2024-04-19T02:03:55.993 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.993 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.993 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.994 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198124+0000", 2024-04-19T02:03:55.994 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0032639430000000001 2024-04-19T02:03:55.994 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.994 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.994 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:55.994 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198217+0000", 2024-04-19T02:03:55.994 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2303e-05 2024-04-19T02:03:55.994 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.994 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.994 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.994 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.223790+0000", 2024-04-19T02:03:55.994 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.025573476000000001 2024-04-19T02:03:55.994 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.994 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.994 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.995 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.995 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.995 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s6 46/26 ECSubWrite(tid=163, reqid=client.4509.0:425, at_version=46'78, trim_to=0'0, roll_forward_to=46'75))", 2024-04-19T02:03:55.995 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:30.198808+0000", 2024-04-19T02:03:55.995 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 25.748526343999998, 2024-04-19T02:03:55.995 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021976591, 2024-04-19T02:03:55.995 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.995 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.995 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:03:55.995 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:55.995 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.995 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:55.995 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198808+0000", 2024-04-19T02:03:55.995 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.996 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.996 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.996 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:55.996 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198808+0000", 2024-04-19T02:03:55.996 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:55.996 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.996 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.996 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:55.996 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198809+0000", 2024-04-19T02:03:55.996 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6330000000000001e-06 2024-04-19T02:03:55.996 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.996 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.996 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:55.996 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198856+0000", 2024-04-19T02:03:55.996 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6414e-05 2024-04-19T02:03:55.997 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.997 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.997 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:55.997 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198882+0000", 2024-04-19T02:03:55.997 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6523999999999999e-05 2024-04-19T02:03:55.997 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.997 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.997 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:55.997 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198901+0000", 2024-04-19T02:03:55.997 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8763e-05 2024-04-19T02:03:55.997 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.997 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.997 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:55.997 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.202201+0000", 2024-04-19T02:03:55.997 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0033002190000000001 2024-04-19T02:03:55.998 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.998 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.998 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:55.998 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.202210+0000", 2024-04-19T02:03:55.998 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7830000000000004e-06 2024-04-19T02:03:55.998 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.998 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.998 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:55.998 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.220681+0000", 2024-04-19T02:03:55.998 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018470594999999999 2024-04-19T02:03:55.998 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.998 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.998 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:55.998 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.220784+0000", 2024-04-19T02:03:55.999 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010365999999999999 2024-04-19T02:03:55.999 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.999 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:55.999 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:55.999 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:55.999 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:55.999 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:501 3.fs0 3:fbe22738:::smithi05930959-7:b [read 0~2883584,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-04-19T02:03:55.999 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:32.048284+0000", 2024-04-19T02:03:55.999 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 23.899050575, 2024-04-19T02:03:55.999 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.02466438, 2024-04-19T02:03:55.999 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:55.999 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:55.999 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:55.999 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:56.000 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:56.000 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:56.000 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 501 2024-04-19T02:03:56.000 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.000 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:56.000 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.000 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:56.000 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048284+0000", 2024-04-19T02:03:56.000 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:56.000 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.000 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.000 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:56.000 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048284+0000", 2024-04-19T02:03:56.000 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:56.000 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.001 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.001 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:56.001 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048285+0000", 2024-04-19T02:03:56.001 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.513e-06 2024-04-19T02:03:56.001 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.001 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.001 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:56.001 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048294+0000", 2024-04-19T02:03:56.001 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4700000000000002e-06 2024-04-19T02:03:56.001 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.001 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.001 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:56.001 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048314+0000", 2024-04-19T02:03:56.001 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0424999999999998e-05 2024-04-19T02:03:56.001 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.002 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.002 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:56.002 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048329+0000", 2024-04-19T02:03:56.002 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5415999999999999e-05 2024-04-19T02:03:56.002 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.002 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.002 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:56.002 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048623+0000", 2024-04-19T02:03:56.002 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00029404700000000003 2024-04-19T02:03:56.002 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.002 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.002 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:03:56.002 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048637+0000", 2024-04-19T02:03:56.002 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3776e-05 2024-04-19T02:03:56.003 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.003 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.003 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:56.003 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.054889+0000", 2024-04-19T02:03:56.003 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.006251876 2024-04-19T02:03:56.003 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.003 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.003 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:56.003 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.055164+0000", 2024-04-19T02:03:56.003 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00027501199999999999 2024-04-19T02:03:56.003 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.003 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.003 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:56.003 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.072948+0000", 2024-04-19T02:03:56.003 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017783845 2024-04-19T02:03:56.004 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:56.004 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:56.004 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:56.005 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.005 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.005 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:502 3.fs0 3:fbe22738:::smithi05930959-7:b [read 0~2883584,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-04-19T02:03:56.005 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:32.048354+0000", 2024-04-19T02:03:56.005 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 23.898980438999999, 2024-04-19T02:03:56.005 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.035084603999999998, 2024-04-19T02:03:56.006 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:56.006 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:56.006 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:56.006 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:56.006 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:56.006 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:56.006 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 502 2024-04-19T02:03:56.006 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.006 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:56.006 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.006 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:56.006 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048354+0000", 2024-04-19T02:03:56.006 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:56.006 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.006 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.007 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:56.007 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048354+0000", 2024-04-19T02:03:56.007 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:56.007 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.007 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.007 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:56.007 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048355+0000", 2024-04-19T02:03:56.007 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1659999999999999e-06 2024-04-19T02:03:56.007 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.007 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.007 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:56.007 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048364+0000", 2024-04-19T02:03:56.007 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5829999999999996e-06 2024-04-19T02:03:56.007 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.007 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.008 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:56.008 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048383+0000", 2024-04-19T02:03:56.008 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8108999999999999e-05 2024-04-19T02:03:56.008 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.008 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.008 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:56.008 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048400+0000", 2024-04-19T02:03:56.008 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7051999999999999e-05 2024-04-19T02:03:56.008 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.008 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.008 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:56.008 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048713+0000", 2024-04-19T02:03:56.008 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00031348900000000002 2024-04-19T02:03:56.008 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.009 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.009 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:03:56.009 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048723+0000", 2024-04-19T02:03:56.009 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6760000000000002e-06 2024-04-19T02:03:56.009 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.009 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.009 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:56.009 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.056039+0000", 2024-04-19T02:03:56.009 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0073158099999999998 2024-04-19T02:03:56.009 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.009 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.009 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:56.009 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.056328+0000", 2024-04-19T02:03:56.009 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028893700000000002 2024-04-19T02:03:56.010 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.010 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.010 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:56.010 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.083438+0000", 2024-04-19T02:03:56.010 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.027110782 2024-04-19T02:03:56.010 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:56.010 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:56.010 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:56.010 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.010 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.010 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:535 3.fs0 3:f8a4544e:::smithi05930959-1:head [copy-from ver 31] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-19T02:03:56.010 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:36.137521+0000", 2024-04-19T02:03:56.010 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 19.809813409, 2024-04-19T02:03:56.010 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.030940776999999999, 2024-04-19T02:03:56.011 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:56.011 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:56.011 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:03:56.011 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:56.011 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:56.011 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:56.011 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 535 2024-04-19T02:03:56.011 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.011 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:56.011 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.011 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:56.011 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137521+0000", 2024-04-19T02:03:56.011 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:56.012 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.012 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.012 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:56.012 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137521+0000", 2024-04-19T02:03:56.012 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:56.012 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.012 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.012 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:56.012 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137524+0000", 2024-04-19T02:03:56.012 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3969999999999998e-06 2024-04-19T02:03:56.012 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.012 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.012 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:56.012 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137540+0000", 2024-04-19T02:03:56.012 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5619000000000001e-05 2024-04-19T02:03:56.013 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.013 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.013 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:56.013 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137571+0000", 2024-04-19T02:03:56.013 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0901000000000001e-05 2024-04-19T02:03:56.013 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.013 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.013 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:56.013 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137591+0000", 2024-04-19T02:03:56.013 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0279e-05 2024-04-19T02:03:56.013 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.013 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.013 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:56.013 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137849+0000", 2024-04-19T02:03:56.013 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000258535 2024-04-19T02:03:56.013 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.014 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.014 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op must wait for map", 2024-04-19T02:03:56.014 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137872+0000", 2024-04-19T02:03:56.014 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2507999999999999e-05 2024-04-19T02:03:56.014 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.014 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.014 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:56.014 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.141172+0000", 2024-04-19T02:03:56.014 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003300234 2024-04-19T02:03:56.014 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.014 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.014 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:56.014 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.141447+0000", 2024-04-19T02:03:56.014 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00027449700000000001 2024-04-19T02:03:56.015 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.015 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.015 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:03:56.015 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.161979+0000", 2024-04-19T02:03:56.015 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020531964 2024-04-19T02:03:56.015 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.015 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.015 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:03:56.015 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.165458+0000", 2024-04-19T02:03:56.015 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0034790060000000002 2024-04-19T02:03:56.015 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.015 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.015 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:03:56.015 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.168324+0000", 2024-04-19T02:03:56.016 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0028665280000000001 2024-04-19T02:03:56.016 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.016 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.016 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:56.016 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.168462+0000", 2024-04-19T02:03:56.016 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013730900000000001 2024-04-19T02:03:56.016 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:56.016 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:56.016 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:56.016 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.016 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.016 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:536 3.fs0 3:f8a4544e:::smithi05930959-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-04-19T02:03:56.016 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:36.137666+0000", 2024-04-19T02:03:56.016 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 19.809668220999999, 2024-04-19T02:03:56.016 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.027514961000000001, 2024-04-19T02:03:56.016 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:56.017 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:56.017 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:56.017 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:56.017 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:56.017 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:56.017 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 536 2024-04-19T02:03:56.017 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.017 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:56.017 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.017 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:56.017 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137666+0000", 2024-04-19T02:03:56.017 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:56.017 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.017 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.017 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:56.018 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137666+0000", 2024-04-19T02:03:56.018 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:56.018 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.018 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.018 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:56.018 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137668+0000", 2024-04-19T02:03:56.018 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9199999999999998e-06 2024-04-19T02:03:56.018 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.018 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.018 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:56.018 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137675+0000", 2024-04-19T02:03:56.018 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.153e-06 2024-04-19T02:03:56.018 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.018 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.019 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:56.019 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137697+0000", 2024-04-19T02:03:56.019 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1522000000000002e-05 2024-04-19T02:03:56.019 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.019 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.019 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:56.019 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137713+0000", 2024-04-19T02:03:56.019 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6016e-05 2024-04-19T02:03:56.019 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.019 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.019 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:56.019 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.138006+0000", 2024-04-19T02:03:56.019 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00029325899999999998 2024-04-19T02:03:56.019 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.019 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.019 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:03:56.020 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.138022+0000", 2024-04-19T02:03:56.020 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6563e-05 2024-04-19T02:03:56.020 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.020 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.020 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:56.020 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.141952+0000", 2024-04-19T02:03:56.020 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039293360000000003 2024-04-19T02:03:56.020 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.020 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.020 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for blocked object", 2024-04-19T02:03:56.020 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.142198+0000", 2024-04-19T02:03:56.020 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00024626500000000001 2024-04-19T02:03:56.020 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.020 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.020 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:56.021 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.164631+0000", 2024-04-19T02:03:56.021 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022432688999999999 2024-04-19T02:03:56.021 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.021 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.021 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:56.021 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.164884+0000", 2024-04-19T02:03:56.021 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000252892 2024-04-19T02:03:56.021 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.021 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.021 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:56.021 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.165181+0000", 2024-04-19T02:03:56.021 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00029734599999999998 2024-04-19T02:03:56.021 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:56.021 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:56.021 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:56.022 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.022 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.022 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:674 3.as0 3:54128c7a:::smithi05930959-20:head [read 0~3194880,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-19T02:03:56.022 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:46.325103+0000", 2024-04-19T02:03:56.022 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.6222307620000009, 2024-04-19T02:03:56.022 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021675204999999999, 2024-04-19T02:03:56.022 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:56.022 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:56.022 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:56.022 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:56.022 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:56.022 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:56.022 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 674 2024-04-19T02:03:56.022 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.023 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:56.023 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.023 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:56.023 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.325103+0000", 2024-04-19T02:03:56.023 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:56.023 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.023 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.023 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:56.023 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.325103+0000", 2024-04-19T02:03:56.023 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:56.023 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.023 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.023 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:56.023 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.325105+0000", 2024-04-19T02:03:56.023 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4670000000000001e-06 2024-04-19T02:03:56.024 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.024 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.024 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:56.024 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.325114+0000", 2024-04-19T02:03:56.024 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0759999999999992e-06 2024-04-19T02:03:56.024 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.024 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.024 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:56.024 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.325134+0000", 2024-04-19T02:03:56.024 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9619e-05 2024-04-19T02:03:56.024 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.024 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.024 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:56.024 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.325148+0000", 2024-04-19T02:03:56.024 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4885999999999999e-05 2024-04-19T02:03:56.025 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.025 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.025 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:56.025 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.327186+0000", 2024-04-19T02:03:56.025 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0020380120000000001 2024-04-19T02:03:56.025 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.025 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.025 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:56.025 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.327378+0000", 2024-04-19T02:03:56.025 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00019130899999999999 2024-04-19T02:03:56.025 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.025 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.025 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:56.025 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.346779+0000", 2024-04-19T02:03:56.025 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019400836000000001 2024-04-19T02:03:56.026 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:56.026 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:56.026 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:56.026 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.026 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.026 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:781 3.as0 3:5a72bad2:::smithi05930959-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e67)", 2024-04-19T02:03:56.026 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:51.156451+0000", 2024-04-19T02:03:56.026 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4.7908835419999996, 2024-04-19T02:03:56.026 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022032399000000001, 2024-04-19T02:03:56.026 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:56.026 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:56.026 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:56.026 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:56.026 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:56.026 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:56.027 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 781 2024-04-19T02:03:56.027 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.027 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:56.027 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.027 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:56.027 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.156451+0000", 2024-04-19T02:03:56.027 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:56.027 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.027 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.027 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:56.027 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.156451+0000", 2024-04-19T02:03:56.027 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:56.027 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.027 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.028 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:56.028 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.156451+0000", 2024-04-19T02:03:56.028 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7700000000000004e-07 2024-04-19T02:03:56.028 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.028 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.028 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:56.028 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.156458+0000", 2024-04-19T02:03:56.028 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9190000000000002e-06 2024-04-19T02:03:56.028 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.028 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.028 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:56.028 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.156492+0000", 2024-04-19T02:03:56.028 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3312000000000002e-05 2024-04-19T02:03:56.028 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.028 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.028 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:56.029 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.156501+0000", 2024-04-19T02:03:56.029 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5200000000000003e-06 2024-04-19T02:03:56.029 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.029 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.029 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:56.029 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.171965+0000", 2024-04-19T02:03:56.029 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015463882999999999 2024-04-19T02:03:56.029 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.029 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.029 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:03:56.029 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.172106+0000", 2024-04-19T02:03:56.029 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014066099999999999 2024-04-19T02:03:56.029 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.029 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.030 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:56.030 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.175964+0000", 2024-04-19T02:03:56.030 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0038586760000000001 2024-04-19T02:03:56.030 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.030 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.030 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:56.030 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.176204+0000", 2024-04-19T02:03:56.030 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00023956699999999999 2024-04-19T02:03:56.030 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.030 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.030 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:56.030 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.178483+0000", 2024-04-19T02:03:56.030 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0022790839999999998 2024-04-19T02:03:56.031 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:56.031 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:56.031 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:56.031 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.031 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.031 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:788 3.as0 3:5a72bad2:::smithi05930959-41:head [read 0~7405568,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-04-19T02:03:56.031 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:54.059369+0000", 2024-04-19T02:03:56.031 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 1.8879647419999999, 2024-04-19T02:03:56.031 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.032465221000000002, 2024-04-19T02:03:56.031 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:56.031 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:56.031 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:56.031 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:56.031 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:56.031 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:56.032 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 788 2024-04-19T02:03:56.032 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.032 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:56.032 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.032 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:56.032 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059369+0000", 2024-04-19T02:03:56.032 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:56.032 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.032 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.032 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:56.032 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059369+0000", 2024-04-19T02:03:56.032 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:56.032 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.032 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.032 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:56.032 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059371+0000", 2024-04-19T02:03:56.033 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.782e-06 2024-04-19T02:03:56.033 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.033 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.033 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:56.033 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059378+0000", 2024-04-19T02:03:56.033 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2899999999999997e-06 2024-04-19T02:03:56.033 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.033 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.033 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:56.033 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059393+0000", 2024-04-19T02:03:56.033 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4792e-05 2024-04-19T02:03:56.033 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.033 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.033 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:56.033 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059403+0000", 2024-04-19T02:03:56.034 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8339999999999994e-06 2024-04-19T02:03:56.034 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.034 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.034 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:56.034 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059476+0000", 2024-04-19T02:03:56.034 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2659999999999999e-05 2024-04-19T02:03:56.034 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.034 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.034 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op must wait for map", 2024-04-19T02:03:56.034 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059482+0000", 2024-04-19T02:03:56.034 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5699999999999998e-06 2024-04-19T02:03:56.034 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.034 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.034 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:56.035 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.062469+0000", 2024-04-19T02:03:56.035 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002987015 2024-04-19T02:03:56.035 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.035 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.035 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:56.035 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.062613+0000", 2024-04-19T02:03:56.035 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014379099999999999 2024-04-19T02:03:56.035 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.035 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.035 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:56.035 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.091835+0000", 2024-04-19T02:03:56.035 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.029221487000000001 2024-04-19T02:03:56.036 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:56.036 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:56.036 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:56.036 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.036 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.036 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:789 3.as0 3:5a72bad2:::smithi05930959-41:head [read 0~7405568,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-04-19T02:03:56.036 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:54.059423+0000", 2024-04-19T02:03:56.036 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 1.887911524, 2024-04-19T02:03:56.036 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.039046966000000002, 2024-04-19T02:03:56.036 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:56.036 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:56.036 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:56.036 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:56.036 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:56.036 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:56.036 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 789 2024-04-19T02:03:56.037 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.037 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:56.037 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.037 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:56.037 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059423+0000", 2024-04-19T02:03:56.037 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:56.037 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.037 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.037 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:56.037 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059423+0000", 2024-04-19T02:03:56.037 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:56.037 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.037 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.037 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:56.037 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059423+0000", 2024-04-19T02:03:56.038 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7999999999999996e-07 2024-04-19T02:03:56.038 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.038 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.038 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:56.038 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059426+0000", 2024-04-19T02:03:56.038 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.275e-06 2024-04-19T02:03:56.038 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.038 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.038 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:56.038 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059433+0000", 2024-04-19T02:03:56.038 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8530000000000003e-06 2024-04-19T02:03:56.038 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.038 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.038 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:56.039 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059439+0000", 2024-04-19T02:03:56.039 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0299999999999999e-06 2024-04-19T02:03:56.039 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.039 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.039 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:56.039 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059570+0000", 2024-04-19T02:03:56.039 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013091000000000001 2024-04-19T02:03:56.039 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.039 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.039 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:03:56.039 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059582+0000", 2024-04-19T02:03:56.039 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1353e-05 2024-04-19T02:03:56.039 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.039 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.039 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:56.039 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.063219+0000", 2024-04-19T02:03:56.040 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0036374979999999999 2024-04-19T02:03:56.040 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.040 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.040 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:56.040 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.063350+0000", 2024-04-19T02:03:56.040 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000130605 2024-04-19T02:03:56.040 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.040 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.040 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:56.040 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.098470+0000", 2024-04-19T02:03:56.040 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.035119961999999998 2024-04-19T02:03:56.040 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:56.040 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:56.040 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:56.041 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.041 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.041 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:790 3.as0 3:5a72bad2:::smithi05930959-41:head [read 0~7405568,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-04-19T02:03:56.041 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:54.059448+0000", 2024-04-19T02:03:56.041 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 1.8878861279999999, 2024-04-19T02:03:56.041 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.045622590999999997, 2024-04-19T02:03:56.041 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:03:56.041 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:03:56.041 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:03:56.041 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:03:56.041 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:03:56.041 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:03:56.041 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 790 2024-04-19T02:03:56.041 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.042 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:03:56.042 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.042 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:03:56.042 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059448+0000", 2024-04-19T02:03:56.042 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:56.042 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.042 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.042 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:03:56.042 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059448+0000", 2024-04-19T02:03:56.042 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:03:56.042 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.042 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.042 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:03:56.042 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059448+0000", 2024-04-19T02:03:56.042 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5999999999999999e-07 2024-04-19T02:03:56.043 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.043 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.043 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:03:56.043 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059451+0000", 2024-04-19T02:03:56.043 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4779999999999998e-06 2024-04-19T02:03:56.043 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.043 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.043 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:03:56.043 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059456+0000", 2024-04-19T02:03:56.043 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7639999999999996e-06 2024-04-19T02:03:56.043 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.043 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.043 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:03:56.043 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059459+0000", 2024-04-19T02:03:56.043 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8319999999999999e-06 2024-04-19T02:03:56.044 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.044 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.044 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:56.044 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059641+0000", 2024-04-19T02:03:56.044 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000181282 2024-04-19T02:03:56.044 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.044 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.044 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:03:56.044 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059651+0000", 2024-04-19T02:03:56.044 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0679000000000001e-05 2024-04-19T02:03:56.044 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.044 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.044 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:03:56.044 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.068040+0000", 2024-04-19T02:03:56.045 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0083886610000000004 2024-04-19T02:03:56.045 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.045 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.045 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:03:56.045 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.068197+0000", 2024-04-19T02:03:56.045 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015667 2024-04-19T02:03:56.045 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:03:56.045 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:03:56.045 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:03:56.045 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.105071+0000", 2024-04-19T02:03:56.045 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.036873864999999999 2024-04-19T02:03:56.045 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:56.045 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:56.045 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:56.046 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:03:56.046 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:03:56.046 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:03:56.047 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:56.049+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:56.048 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-19T02:03:56.085 INFO:tasks.rados.rados.0.smithi059.stdout:309: done (0 left) 2024-04-19T02:03:56.085 INFO:tasks.rados.rados.0.smithi059.stdout:310: snap_remove snap 19 2024-04-19T02:03:56.086 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-04-19T02:03:56.086 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T02:03:56.086 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-19T02:03:56.104 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-04-19T02:03:56.104 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T02:03:56.122 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-04-19T02:03:56.122 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T02:03:56.148 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:56.149+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:56.156 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-04-19T02:03:56.156 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T02:03:56.172 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:56.172 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:56.187 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-19T02:03:56.193 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-04-19T02:03:56.194 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T02:03:56.229 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-04-19T02:03:56.229 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T02:03:56.250 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:56.253+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:56.253 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-04-19T02:03:56.253 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T02:03:56.274 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-04-19T02:03:56.275 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T02:03:56.296 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-04-19T02:03:56.296 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T02:03:56.317 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-04-19T02:03:56.318 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T02:03:56.328 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:56.328 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:56.338 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-04-19T02:03:56.338 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T02:03:56.342 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-19T02:03:56.359 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-04-19T02:03:56.359 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T02:03:56.360 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:56.361+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:56.452 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:56.453+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:56.476 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:56.476 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:56.490 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-19T02:03:56.553 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:56.553+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:56.618 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:56.618 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:56.632 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-19T02:03:56.679 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:56.681+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:56.755 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:03:56.757+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:03:56.761 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:56.761 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:56.777 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-19T02:03:56.863 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:56.865+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:56.945 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:56.946 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:56.957 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:56.957+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:56.959 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-19T02:03:57.047 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is set 2024-04-19T02:03:57.050 INFO:tasks.rados.rados.0.smithi059.stdout:310: done (0 left) 2024-04-19T02:03:57.050 INFO:tasks.rados.rados.0.smithi059.stdout:311: rollback oid 38 current snap is 25 2024-04-19T02:03:57.051 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 38 to 10 2024-04-19T02:03:57.051 INFO:tasks.rados.rados.0.smithi059.stdout:312: rollback oid 30 current snap is 25 2024-04-19T02:03:57.051 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 30 to 21 2024-04-19T02:03:57.051 INFO:tasks.rados.rados.0.smithi059.stdout:313: copy_from oid 23 from oid 20 current snap is 25 2024-04-19T02:03:57.051 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 48 current snap is 25 2024-04-19T02:03:57.051 INFO:tasks.rados.rados.0.smithi059.stdout:314: seq_num 146 ranges {2703360=49152} 2024-04-19T02:03:57.064 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:57.065+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:57.070 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-19T02:03:57.070 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-19T02:03:57.070 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-04-19T02:03:57.070 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-19T02:03:57.070 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-19T02:03:57.071 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2024-04-19T02:03:57.076 INFO:tasks.rados.rados.0.smithi059.stdout:314: writing smithi05930959-48 from 2703360 to 2752512 tid 1 2024-04-19T02:03:57.077 INFO:tasks.rados.rados.0.smithi059.stdout:315: read oid 9 snap -1 2024-04-19T02:03:57.077 INFO:tasks.rados.rados.0.smithi059.stdout:315: expect deleted 2024-04-19T02:03:57.077 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 36 current snap is 25 2024-04-19T02:03:57.077 INFO:tasks.rados.rados.0.smithi059.stdout:316: seq_num 147 ranges {3637248=65536} 2024-04-19T02:03:57.104 INFO:tasks.rados.rados.0.smithi059.stdout:316: writing smithi05930959-36 from 3637248 to 3702784 tid 1 2024-04-19T02:03:57.104 INFO:tasks.rados.rados.0.smithi059.stdout:311: finishing rollback tid 1 to smithi05930959-38 2024-04-19T02:03:57.104 INFO:tasks.rados.rados.0.smithi059.stdout:312: finishing rollback tid 0 to smithi05930959-30 2024-04-19T02:03:57.104 INFO:tasks.rados.rados.0.smithi059.stdout:311: finishing rollback tid 2 to smithi05930959-38 2024-04-19T02:03:57.104 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 38 v 83 (ObjNum 83 snap 8 seq_num 83) dirty exists 2024-04-19T02:03:57.104 INFO:tasks.rados.rados.0.smithi059.stdout:312: finishing rollback tid 1 to smithi05930959-30 2024-04-19T02:03:57.104 INFO:tasks.rados.rados.0.smithi059.stdout:312: finishing rollback tid 2 to smithi05930959-30 2024-04-19T02:03:57.104 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 30 v 54 (ObjNum 121 snap 19 seq_num 121) dirty exists 2024-04-19T02:03:57.104 INFO:tasks.rados.rados.0.smithi059.stdout:311: done (5 left) 2024-04-19T02:03:57.105 INFO:tasks.rados.rados.0.smithi059.stdout:312: done (4 left) 2024-04-19T02:03:57.105 INFO:tasks.rados.rados.0.smithi059.stdout:315: done (3 left) 2024-04-19T02:03:57.105 INFO:tasks.rados.rados.0.smithi059.stdout:317: snap_create 2024-04-19T02:03:57.108 INFO:tasks.rados.rados.0.smithi059.stdout:314: finishing write tid 1 to smithi05930959-48 2024-04-19T02:03:57.110 INFO:tasks.rados.rados.0.smithi059.stdout:314: finishing write tid 2 to smithi05930959-48 2024-04-19T02:03:57.115 INFO:tasks.rados.rados.0.smithi059.stdout:316: finishing write tid 1 to smithi05930959-36 2024-04-19T02:03:57.116 INFO:tasks.rados.rados.0.smithi059.stdout:316: finishing write tid 2 to smithi05930959-36 2024-04-19T02:03:57.116 INFO:tasks.rados.rados.0.smithi059.stdout:313: finishing copy_from racing read to smithi05930959-23 2024-04-19T02:03:57.118 INFO:tasks.rados.rados.0.smithi059.stdout:314: finishing write tid 3 to smithi05930959-48 2024-04-19T02:03:57.118 INFO:tasks.rados.rados.0.smithi059.stdout:314: oid 48 updating version 0 to 83 2024-04-19T02:03:57.118 INFO:tasks.rados.rados.0.smithi059.stdout:314: oid 48 updating version 83 to 84 2024-04-19T02:03:57.118 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 48 v 84 (ObjNum 146 snap 25 seq_num 146) dirty exists 2024-04-19T02:03:57.119 INFO:tasks.rados.rados.0.smithi059.stdout:314: left oid 48 (ObjNum 146 snap 25 seq_num 146) 2024-04-19T02:03:57.119 INFO:tasks.rados.rados.0.smithi059.stdout:316: finishing write tid 3 to smithi05930959-36 2024-04-19T02:03:57.119 INFO:tasks.rados.rados.0.smithi059.stdout:316: oid 36 updating version 0 to 32 2024-04-19T02:03:57.119 INFO:tasks.rados.rados.0.smithi059.stdout:316: oid 36 updating version 32 to 33 2024-04-19T02:03:57.119 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 36 v 33 (ObjNum 147 snap 25 seq_num 147) dirty exists 2024-04-19T02:03:57.119 INFO:tasks.rados.rados.0.smithi059.stdout:316: left oid 36 (ObjNum 147 snap 25 seq_num 147) 2024-04-19T02:03:57.121 INFO:tasks.rados.rados.0.smithi059.stdout:313: finishing copy_from to smithi05930959-23 2024-04-19T02:03:57.121 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 23 v 64 (ObjNum 94 snap 9 seq_num 94) dirty exists 2024-04-19T02:03:57.159 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:57.161+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:57.165 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:57.165 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:57.181 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-19T02:03:57.215 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:03:57.215 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-19T02:03:57.215 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:03:57.275 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:57.277+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:57.340 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:57.341 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:57.354 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-19T02:03:57.361 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:57.361+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:57.462 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:57.461+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:57.528 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:57.528 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:57.542 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-19T02:03:57.563 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:57.565+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:57.663 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:57.665+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:57.709 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:57.710 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:57.724 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-19T02:03:57.767 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:57.769+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:57.865 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:57.865+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:57.893 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:57.893 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:57.907 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-19T02:03:57.966 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:57.969+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:58.070 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:58.069+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:58.071 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:58.071 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:58.085 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-19T02:03:58.111 INFO:tasks.rados.rados.0.smithi059.stdout:313: done (3 left) 2024-04-19T02:03:58.111 INFO:tasks.rados.rados.0.smithi059.stdout:314: done (2 left) 2024-04-19T02:03:58.111 INFO:tasks.rados.rados.0.smithi059.stdout:316: done (1 left) 2024-04-19T02:03:58.111 INFO:tasks.rados.rados.0.smithi059.stdout:317: done (0 left) 2024-04-19T02:03:58.111 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 4 current snap is 26 2024-04-19T02:03:58.112 INFO:tasks.rados.rados.0.smithi059.stdout:318: seq_num 148 ranges {3817472=81920} 2024-04-19T02:03:58.143 INFO:tasks.rados.rados.0.smithi059.stdout:318: writing smithi05930959-4 from 3817472 to 3899392 tid 1 2024-04-19T02:03:58.144 INFO:tasks.rados.rados.0.smithi059.stdout:319: rmattr oid 10 current snap is 26 2024-04-19T02:03:58.144 INFO:tasks.rados.rados.0.smithi059.stdout:319: done (1 left) 2024-04-19T02:03:58.144 INFO:tasks.rados.rados.0.smithi059.stdout:320: setattr oid 11 current snap is 26 2024-04-19T02:03:58.148 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 14 current snap is 26 2024-04-19T02:03:58.148 INFO:tasks.rados.rados.0.smithi059.stdout:321: seq_num 149 ranges {2359296=98304} 2024-04-19T02:03:58.168 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:58.169+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:58.169 INFO:tasks.rados.rados.0.smithi059.stdout:321: writing smithi05930959-14 from 2359296 to 2457600 tid 1 2024-04-19T02:03:58.169 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 46 current snap is 26 2024-04-19T02:03:58.169 INFO:tasks.rados.rados.0.smithi059.stdout:322: seq_num 150 ranges {4243456=114688} 2024-04-19T02:03:58.200 INFO:tasks.rados.rados.0.smithi059.stdout:322: writing smithi05930959-46 from 4243456 to 4358144 tid 1 2024-04-19T02:03:58.200 INFO:tasks.rados.rados.0.smithi059.stdout:323: read oid 42 snap -1 2024-04-19T02:03:58.200 INFO:tasks.rados.rados.0.smithi059.stdout:323: expect (ObjNum 117 snap 17 seq_num 117) 2024-04-19T02:03:58.200 INFO:tasks.rados.rados.0.smithi059.stdout:324: snap_create 2024-04-19T02:03:58.200 INFO:tasks.rados.rados.0.smithi059.stdout:318: finishing write tid 1 to smithi05930959-4 2024-04-19T02:03:58.200 INFO:tasks.rados.rados.0.smithi059.stdout:318: finishing write tid 2 to smithi05930959-4 2024-04-19T02:03:58.200 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 11 v 7 (ObjNum 33 snap 0 seq_num 948807216) dirty exists 2024-04-19T02:03:58.200 INFO:tasks.rados.rados.0.smithi059.stdout:318: finishing write tid 3 to smithi05930959-4 2024-04-19T02:03:58.200 INFO:tasks.rados.rados.0.smithi059.stdout:318: oid 4 updating version 0 to 87 2024-04-19T02:03:58.200 INFO:tasks.rados.rados.0.smithi059.stdout:318: oid 4 updating version 87 to 88 2024-04-19T02:03:58.200 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 4 v 88 (ObjNum 148 snap 26 seq_num 148) dirty exists 2024-04-19T02:03:58.201 INFO:tasks.rados.rados.0.smithi059.stdout:318: left oid 4 (ObjNum 148 snap 26 seq_num 148) 2024-04-19T02:03:58.201 INFO:tasks.rados.rados.0.smithi059.stdout:321: finishing write tid 1 to smithi05930959-14 2024-04-19T02:03:58.201 INFO:tasks.rados.rados.0.smithi059.stdout:321: finishing write tid 2 to smithi05930959-14 2024-04-19T02:03:58.201 INFO:tasks.rados.rados.0.smithi059.stdout:321: finishing write tid 3 to smithi05930959-14 2024-04-19T02:03:58.201 INFO:tasks.rados.rados.0.smithi059.stdout:321: oid 14 updating version 0 to 91 2024-04-19T02:03:58.201 INFO:tasks.rados.rados.0.smithi059.stdout:321: oid 14 version 91 is already newer than 90 2024-04-19T02:03:58.201 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 14 v 91 (ObjNum 149 snap 26 seq_num 149) dirty exists 2024-04-19T02:03:58.201 INFO:tasks.rados.rados.0.smithi059.stdout:321: left oid 14 (ObjNum 149 snap 26 seq_num 149) 2024-04-19T02:03:58.213 INFO:tasks.rados.rados.0.smithi059.stdout:322: finishing write tid 1 to smithi05930959-46 2024-04-19T02:03:58.216 INFO:tasks.rados.rados.0.smithi059.stdout:322: finishing write tid 2 to smithi05930959-46 2024-04-19T02:03:58.222 INFO:tasks.rados.rados.0.smithi059.stdout:322: finishing write tid 3 to smithi05930959-46 2024-04-19T02:03:58.222 INFO:tasks.rados.rados.0.smithi059.stdout:322: oid 46 updating version 0 to 27 2024-04-19T02:03:58.222 INFO:tasks.rados.rados.0.smithi059.stdout:322: oid 46 version 27 is already newer than 26 2024-04-19T02:03:58.222 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 46 v 27 (ObjNum 150 snap 26 seq_num 150) dirty exists 2024-04-19T02:03:58.222 INFO:tasks.rados.rados.0.smithi059.stdout:322: left oid 46 (ObjNum 150 snap 26 seq_num 150) 2024-04-19T02:03:58.226 INFO:tasks.rados.rados.0.smithi059.stdout:323: expect (ObjNum 117 snap 17 seq_num 117) 2024-04-19T02:03:58.228 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:58.228 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:58.242 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-19T02:03:58.269 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:03:58.269+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:03:58.370 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:03:58.369+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:03:58.378 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T02:03:58.409 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:58.409 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:58.423 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-19T02:03:58.511 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:03:58.513+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:03:58.527 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-04-19T02:03:58.527 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T02:03:58.544 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-04-19T02:03:58.544 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T02:03:58.560 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-04-19T02:03:58.561 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T02:03:58.572 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:03:58.573+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:03:58.578 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-04-19T02:03:58.578 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T02:03:58.592 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:03:58.592 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:03:58.598 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-04-19T02:03:58.599 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T02:03:58.607 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-19T02:03:58.618 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-04-19T02:03:58.619 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T02:03:58.637 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-04-19T02:03:58.637 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T02:03:58.657 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-04-19T02:03:58.657 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T02:03:58.674 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:58.677+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:58.678 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-04-19T02:03:58.678 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T02:03:58.700 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-04-19T02:03:58.701 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T02:03:58.723 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-04-19T02:03:58.723 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T02:03:58.746 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-04-19T02:03:58.746 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T02:03:58.774 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:58.773+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:58.775 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:58.775 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:58.776 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 0 2024-04-19T02:03:58.776 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:58.789 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-19T02:03:58.875 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:58.877+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:58.955 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:58.955 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:58.955 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-04-19T02:03:58.955 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-04-19T02:03:58.955 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:58.969 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-19T02:03:58.976 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:58.977+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:59.077 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:03:59.077+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:03:59.082 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-19T02:03:59.096 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:59.096 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-04-19T02:03:59.096 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-04-19T02:03:59.096 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-04-19T02:03:59.096 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.097 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.4s1 42/26 ECSubWrite(tid=44, reqid=osd.1.0:44, at_version=42'32, trim_to=0'0, roll_forward_to=40'28))", 2024-04-19T02:03:59.097 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.738509+0000", 2024-04-19T02:03:59.097 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 32.357557088, 2024-04-19T02:03:59.097 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.01226059, 2024-04-19T02:03:59.097 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.097 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.097 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.097 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.097 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.097 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.097 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.738509+0000", 2024-04-19T02:03:59.097 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.098 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.098 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.098 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.098 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.738509+0000", 2024-04-19T02:03:59.098 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.098 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.098 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.098 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.098 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.738511+0000", 2024-04-19T02:03:59.098 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.84e-06 2024-04-19T02:03:59.098 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.098 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.098 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.098 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.738565+0000", 2024-04-19T02:03:59.099 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3643000000000001e-05 2024-04-19T02:03:59.099 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.099 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.099 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.099 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.738593+0000", 2024-04-19T02:03:59.099 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8810999999999999e-05 2024-04-19T02:03:59.099 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.099 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.099 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.099 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.738615+0000", 2024-04-19T02:03:59.099 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1421999999999999e-05 2024-04-19T02:03:59.099 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.099 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.100 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.100 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.739435+0000", 2024-04-19T02:03:59.100 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00081972100000000003 2024-04-19T02:03:59.100 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.100 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.100 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.100 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.739460+0000", 2024-04-19T02:03:59.100 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5646000000000001e-05 2024-04-19T02:03:59.100 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.100 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.100 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.100 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.750672+0000", 2024-04-19T02:03:59.100 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011211563000000001 2024-04-19T02:03:59.100 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.101 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.101 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.101 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.750770+0000", 2024-04-19T02:03:59.101 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7943999999999996e-05 2024-04-19T02:03:59.101 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.101 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.101 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.101 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.101 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.101 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.ds4 42/26 ECSubWrite(tid=163, reqid=osd.8.0:163, at_version=42'14, trim_to=0'0, roll_forward_to=39'12))", 2024-04-19T02:03:59.101 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.742684+0000", 2024-04-19T02:03:59.101 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 32.353382140999997, 2024-04-19T02:03:59.101 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012960826, 2024-04-19T02:03:59.101 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.102 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.102 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.102 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.102 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.102 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.102 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.742684+0000", 2024-04-19T02:03:59.102 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.102 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.102 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.102 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.102 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.742684+0000", 2024-04-19T02:03:59.102 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.102 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.103 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.103 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.103 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.742685+0000", 2024-04-19T02:03:59.103 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0999999999999998e-07 2024-04-19T02:03:59.103 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.103 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.103 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.103 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.742739+0000", 2024-04-19T02:03:59.103 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4089999999999999e-05 2024-04-19T02:03:59.103 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.103 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.103 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.103 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.742770+0000", 2024-04-19T02:03:59.103 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0580999999999997e-05 2024-04-19T02:03:59.103 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.104 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.104 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.104 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.742792+0000", 2024-04-19T02:03:59.104 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2985000000000001e-05 2024-04-19T02:03:59.104 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.104 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.104 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.104 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.743203+0000", 2024-04-19T02:03:59.104 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000410379 2024-04-19T02:03:59.104 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.104 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.104 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.104 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.743230+0000", 2024-04-19T02:03:59.104 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6848000000000001e-05 2024-04-19T02:03:59.105 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.105 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.105 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.105 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755545+0000", 2024-04-19T02:03:59.105 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012315609 2024-04-19T02:03:59.105 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.105 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.105 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.105 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755645+0000", 2024-04-19T02:03:59.105 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9623999999999993e-05 2024-04-19T02:03:59.105 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.105 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.105 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.106 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.106 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.106 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.2s1 42/26 ECSubWrite(tid=132, reqid=osd.3.0:132, at_version=42'63, trim_to=0'0, roll_forward_to=40'61))", 2024-04-19T02:03:59.106 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.746263+0000", 2024-04-19T02:03:59.106 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 32.349803276999999, 2024-04-19T02:03:59.106 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013143470000000001, 2024-04-19T02:03:59.106 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.106 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.106 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.106 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.106 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.106 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.106 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746263+0000", 2024-04-19T02:03:59.106 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.107 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.107 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.107 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.107 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746263+0000", 2024-04-19T02:03:59.107 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.107 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.107 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.107 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.107 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746264+0000", 2024-04-19T02:03:59.107 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9399999999999993e-07 2024-04-19T02:03:59.107 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.107 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.107 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.107 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746323+0000", 2024-04-19T02:03:59.107 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8686e-05 2024-04-19T02:03:59.107 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.108 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.108 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.108 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746353+0000", 2024-04-19T02:03:59.108 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0394000000000001e-05 2024-04-19T02:03:59.108 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.108 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.108 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.108 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746375+0000", 2024-04-19T02:03:59.108 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1934999999999999e-05 2024-04-19T02:03:59.108 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.108 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.108 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.108 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746786+0000", 2024-04-19T02:03:59.108 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00041093199999999997 2024-04-19T02:03:59.108 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.108 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.109 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.109 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746812+0000", 2024-04-19T02:03:59.109 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6392000000000001e-05 2024-04-19T02:03:59.109 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.109 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.109 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.109 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.759279+0000", 2024-04-19T02:03:59.109 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012466589 2024-04-19T02:03:59.109 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.109 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.109 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.109 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.759406+0000", 2024-04-19T02:03:59.109 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012754800000000001 2024-04-19T02:03:59.109 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.109 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.109 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.110 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.110 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.110 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.8s4 42/26 ECSubWrite(tid=71, reqid=osd.4.0:71, at_version=42'33, trim_to=0'0, roll_forward_to=37'31))", 2024-04-19T02:03:59.110 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.746410+0000", 2024-04-19T02:03:59.110 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 32.349656273000001, 2024-04-19T02:03:59.110 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012849090000000001, 2024-04-19T02:03:59.110 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.110 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.110 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.110 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.110 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.110 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.110 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746410+0000", 2024-04-19T02:03:59.111 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.111 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.111 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.111 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.111 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746410+0000", 2024-04-19T02:03:59.111 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.111 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.111 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.111 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.111 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746411+0000", 2024-04-19T02:03:59.111 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9000000000000005e-07 2024-04-19T02:03:59.111 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.111 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.111 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.111 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746467+0000", 2024-04-19T02:03:59.111 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6149000000000001e-05 2024-04-19T02:03:59.111 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.112 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.112 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.112 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746511+0000", 2024-04-19T02:03:59.112 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4317000000000003e-05 2024-04-19T02:03:59.112 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.112 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.112 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.112 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746535+0000", 2024-04-19T02:03:59.112 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3655e-05 2024-04-19T02:03:59.112 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.112 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.112 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.112 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.747876+0000", 2024-04-19T02:03:59.112 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0013412490000000001 2024-04-19T02:03:59.112 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.112 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.113 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.113 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.747901+0000", 2024-04-19T02:03:59.113 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4572000000000001e-05 2024-04-19T02:03:59.113 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.113 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.113 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.113 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.759159+0000", 2024-04-19T02:03:59.113 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011258351 2024-04-19T02:03:59.113 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.113 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.113 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.113 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.759259+0000", 2024-04-19T02:03:59.113 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9807000000000002e-05 2024-04-19T02:03:59.113 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.113 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.113 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.114 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.114 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.114 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.as4 42/26 ECSubWrite(tid=82, reqid=osd.11.0:82, at_version=42'34, trim_to=0'0, roll_forward_to=39'32))", 2024-04-19T02:03:59.114 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.749301+0000", 2024-04-19T02:03:59.114 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 32.346765066000003, 2024-04-19T02:03:59.114 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.015619305999999999, 2024-04-19T02:03:59.114 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.114 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.114 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.114 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.114 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.114 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.114 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749301+0000", 2024-04-19T02:03:59.114 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.115 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.115 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.115 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.115 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749301+0000", 2024-04-19T02:03:59.115 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.115 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.115 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.115 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.115 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749302+0000", 2024-04-19T02:03:59.115 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5000000000000002e-07 2024-04-19T02:03:59.115 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.115 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.116 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.116 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749357+0000", 2024-04-19T02:03:59.116 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5013000000000003e-05 2024-04-19T02:03:59.116 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.116 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.116 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.116 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749391+0000", 2024-04-19T02:03:59.116 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3821e-05 2024-04-19T02:03:59.116 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.116 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.116 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.116 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749414+0000", 2024-04-19T02:03:59.116 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3584999999999999e-05 2024-04-19T02:03:59.116 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.117 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.117 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.117 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.750335+0000", 2024-04-19T02:03:59.117 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00092032799999999999 2024-04-19T02:03:59.117 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.117 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.117 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.117 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.750362+0000", 2024-04-19T02:03:59.117 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7217999999999999e-05 2024-04-19T02:03:59.117 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.117 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.117 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.118 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.764821+0000", 2024-04-19T02:03:59.118 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014459520999999999 2024-04-19T02:03:59.118 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.118 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.118 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.118 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.764921+0000", 2024-04-19T02:03:59.118 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9170000000000001e-05 2024-04-19T02:03:59.118 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.118 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.118 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.118 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.118 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.119 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.5s7 42/26 ECSubWrite(tid=51, reqid=osd.10.0:51, at_version=42'27, trim_to=0'0, roll_forward_to=39'25))", 2024-04-19T02:03:59.119 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.749382+0000", 2024-04-19T02:03:59.119 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 32.346683828000003, 2024-04-19T02:03:59.119 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010066832, 2024-04-19T02:03:59.119 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.119 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.119 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.119 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.119 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.119 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.119 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749382+0000", 2024-04-19T02:03:59.119 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.119 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.119 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.120 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.120 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749382+0000", 2024-04-19T02:03:59.120 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.120 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.120 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.120 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.120 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749383+0000", 2024-04-19T02:03:59.120 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7000000000000003e-07 2024-04-19T02:03:59.120 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.120 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.120 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.120 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749433+0000", 2024-04-19T02:03:59.120 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9812999999999999e-05 2024-04-19T02:03:59.121 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.121 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.121 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.121 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749459+0000", 2024-04-19T02:03:59.121 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5565e-05 2024-04-19T02:03:59.121 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.121 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.121 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.121 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749478+0000", 2024-04-19T02:03:59.121 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9216e-05 2024-04-19T02:03:59.121 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.121 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.121 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.121 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749657+0000", 2024-04-19T02:03:59.121 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017872799999999999 2024-04-19T02:03:59.121 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.122 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.122 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.122 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749682+0000", 2024-04-19T02:03:59.122 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5734000000000001e-05 2024-04-19T02:03:59.122 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.122 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.122 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.122 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.759360+0000", 2024-04-19T02:03:59.122 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0096778550000000008 2024-04-19T02:03:59.122 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.122 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.122 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.122 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.759449+0000", 2024-04-19T02:03:59.122 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8950999999999996e-05 2024-04-19T02:03:59.123 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.123 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.123 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.123 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.123 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.123 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.2s1 42/26 ECSubWrite(tid=134, reqid=osd.3.0:134, at_version=42'65, trim_to=0'0, roll_forward_to=40'61))", 2024-04-19T02:03:59.123 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.749777+0000", 2024-04-19T02:03:59.123 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 32.346289263999999, 2024-04-19T02:03:59.123 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.015233401000000001, 2024-04-19T02:03:59.123 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.123 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.123 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.123 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.124 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.124 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.124 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749777+0000", 2024-04-19T02:03:59.124 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.124 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.124 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.124 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.124 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749777+0000", 2024-04-19T02:03:59.124 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.124 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.124 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.124 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.124 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749778+0000", 2024-04-19T02:03:59.124 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.336e-06 2024-04-19T02:03:59.125 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.125 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.125 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.125 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749823+0000", 2024-04-19T02:03:59.125 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4277000000000003e-05 2024-04-19T02:03:59.125 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.125 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.125 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.125 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749845+0000", 2024-04-19T02:03:59.125 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2677999999999999e-05 2024-04-19T02:03:59.125 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.125 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.125 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.125 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749864+0000", 2024-04-19T02:03:59.125 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8913e-05 2024-04-19T02:03:59.126 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.126 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.126 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.126 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.753987+0000", 2024-04-19T02:03:59.126 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0041232070000000003 2024-04-19T02:03:59.126 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.126 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.126 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.126 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.754010+0000", 2024-04-19T02:03:59.126 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2792000000000001e-05 2024-04-19T02:03:59.126 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.126 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.127 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.127 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.764933+0000", 2024-04-19T02:03:59.127 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.01092277 2024-04-19T02:03:59.127 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.127 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.127 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.127 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.765010+0000", 2024-04-19T02:03:59.127 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7428000000000005e-05 2024-04-19T02:03:59.127 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.127 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.127 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.127 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.127 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.127 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.es2 42/26 ECSubWrite(tid=32, reqid=osd.7.0:32, at_version=42'25, trim_to=0'0, roll_forward_to=40'23))", 2024-04-19T02:03:59.127 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.752167+0000", 2024-04-19T02:03:59.128 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 32.343899280000002, 2024-04-19T02:03:59.128 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012865329, 2024-04-19T02:03:59.128 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.128 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.128 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.128 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.128 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.128 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.128 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.752167+0000", 2024-04-19T02:03:59.128 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.128 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.128 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.128 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.128 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.752167+0000", 2024-04-19T02:03:59.128 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.129 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.129 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.129 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.129 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.752168+0000", 2024-04-19T02:03:59.129 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6700000000000003e-07 2024-04-19T02:03:59.129 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.129 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.129 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.129 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.752240+0000", 2024-04-19T02:03:59.129 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1988000000000003e-05 2024-04-19T02:03:59.129 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.129 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.129 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.129 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.752265+0000", 2024-04-19T02:03:59.129 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5626000000000002e-05 2024-04-19T02:03:59.130 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.130 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.130 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.130 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.752287+0000", 2024-04-19T02:03:59.130 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2045e-05 2024-04-19T02:03:59.130 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.130 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.130 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.130 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.752546+0000", 2024-04-19T02:03:59.130 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000258342 2024-04-19T02:03:59.130 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.130 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.130 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.130 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.752571+0000", 2024-04-19T02:03:59.131 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4941999999999999e-05 2024-04-19T02:03:59.131 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.131 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.131 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.131 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.764965+0000", 2024-04-19T02:03:59.131 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012394196999999999 2024-04-19T02:03:59.131 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.131 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.131 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.131 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.765032+0000", 2024-04-19T02:03:59.131 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7621999999999997e-05 2024-04-19T02:03:59.131 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.131 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.131 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.131 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.131 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.132 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.4s1 42/26 ECSubWrite(tid=45, reqid=osd.1.0:44, at_version=0'0, trim_to=0'0, roll_forward_to=42'32))", 2024-04-19T02:03:59.132 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.754514+0000", 2024-04-19T02:03:59.132 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 32.341552585000002, 2024-04-19T02:03:59.132 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011039706, 2024-04-19T02:03:59.132 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.132 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.132 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.132 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.132 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.132 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.132 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.754514+0000", 2024-04-19T02:03:59.132 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.132 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.132 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.132 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.132 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.754514+0000", 2024-04-19T02:03:59.133 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.133 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.133 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.133 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.133 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.754516+0000", 2024-04-19T02:03:59.133 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8560000000000001e-06 2024-04-19T02:03:59.133 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.133 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.133 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.133 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.754544+0000", 2024-04-19T02:03:59.133 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8248000000000001e-05 2024-04-19T02:03:59.133 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.133 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.133 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.133 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.754574+0000", 2024-04-19T02:03:59.134 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9935000000000001e-05 2024-04-19T02:03:59.134 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.134 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.134 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.134 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.754595+0000", 2024-04-19T02:03:59.134 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1596000000000001e-05 2024-04-19T02:03:59.134 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.134 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.134 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.134 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.754758+0000", 2024-04-19T02:03:59.134 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016278500000000001 2024-04-19T02:03:59.134 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.134 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.134 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.134 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.754784+0000", 2024-04-19T02:03:59.135 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5698e-05 2024-04-19T02:03:59.135 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.135 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.135 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.135 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.765464+0000", 2024-04-19T02:03:59.135 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010680044 2024-04-19T02:03:59.135 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.135 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.135 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.135 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.765553+0000", 2024-04-19T02:03:59.135 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.9543999999999995e-05 2024-04-19T02:03:59.135 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.135 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.135 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.135 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.135 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.136 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.fs4 42/26 ECSubWrite(tid=83, reqid=osd.11.0:83, at_version=42'26, trim_to=0'0, roll_forward_to=39'24))", 2024-04-19T02:03:59.136 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.755206+0000", 2024-04-19T02:03:59.136 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 32.340860071999998, 2024-04-19T02:03:59.136 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0097280309999999998, 2024-04-19T02:03:59.136 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.136 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.136 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.136 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.136 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.136 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.136 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755206+0000", 2024-04-19T02:03:59.136 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.136 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.136 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.137 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.137 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755206+0000", 2024-04-19T02:03:59.137 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.137 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.137 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.137 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.137 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755208+0000", 2024-04-19T02:03:59.137 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9400000000000001e-06 2024-04-19T02:03:59.137 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.137 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.137 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.137 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755258+0000", 2024-04-19T02:03:59.137 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0040000000000002e-05 2024-04-19T02:03:59.137 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.137 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.138 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.138 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755289+0000", 2024-04-19T02:03:59.138 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0368000000000001e-05 2024-04-19T02:03:59.138 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.138 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.138 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.138 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755311+0000", 2024-04-19T02:03:59.138 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2430999999999999e-05 2024-04-19T02:03:59.138 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.138 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.138 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.138 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755525+0000", 2024-04-19T02:03:59.138 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000214513 2024-04-19T02:03:59.138 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.139 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.139 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.139 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755550+0000", 2024-04-19T02:03:59.139 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4754999999999999e-05 2024-04-19T02:03:59.139 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.139 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.139 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.139 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.764835+0000", 2024-04-19T02:03:59.139 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.009284647 2024-04-19T02:03:59.139 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.139 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.139 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.139 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.764934+0000", 2024-04-19T02:03:59.139 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9337000000000001e-05 2024-04-19T02:03:59.139 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.140 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.140 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.140 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.140 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.140 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.ds4 46/26 ECSubWrite(tid=187, reqid=osd.8.0:187, at_version=46'35, trim_to=0'0, roll_forward_to=45'31))", 2024-04-19T02:03:59.140 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.058841+0000", 2024-04-19T02:03:59.140 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 29.037225121999999, 2024-04-19T02:03:59.140 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0093001779999999992, 2024-04-19T02:03:59.140 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.140 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.140 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.140 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.140 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.140 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.140 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.058841+0000", 2024-04-19T02:03:59.141 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.141 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.141 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.141 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.141 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.058841+0000", 2024-04-19T02:03:59.141 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.141 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.141 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.141 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.141 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.058843+0000", 2024-04-19T02:03:59.141 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.81e-06 2024-04-19T02:03:59.141 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.141 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.141 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.141 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.058892+0000", 2024-04-19T02:03:59.142 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9333999999999999e-05 2024-04-19T02:03:59.142 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.142 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.142 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.142 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.058927+0000", 2024-04-19T02:03:59.142 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4357999999999999e-05 2024-04-19T02:03:59.142 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.142 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.142 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.142 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.058950+0000", 2024-04-19T02:03:59.142 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3261e-05 2024-04-19T02:03:59.142 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.142 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.143 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.143 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.060130+0000", 2024-04-19T02:03:59.143 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0011804249999999999 2024-04-19T02:03:59.143 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.143 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.143 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.143 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.060154+0000", 2024-04-19T02:03:59.143 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4111999999999999e-05 2024-04-19T02:03:59.143 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.143 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.143 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.143 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.068068+0000", 2024-04-19T02:03:59.143 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0079134419999999997 2024-04-19T02:03:59.143 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.144 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.144 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.144 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.068141+0000", 2024-04-19T02:03:59.144 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3436000000000006e-05 2024-04-19T02:03:59.144 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.144 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.144 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.144 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.144 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.144 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.as4 46/26 ECSubWrite(tid=103, reqid=osd.11.0:103, at_version=46'49, trim_to=0'0, roll_forward_to=45'45))", 2024-04-19T02:03:59.144 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.067936+0000", 2024-04-19T02:03:59.144 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 29.028129878000001, 2024-04-19T02:03:59.144 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.017806114000000001, 2024-04-19T02:03:59.144 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.145 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.145 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.145 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.145 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.145 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.145 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.067936+0000", 2024-04-19T02:03:59.145 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.145 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.145 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.145 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.145 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.067936+0000", 2024-04-19T02:03:59.145 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.145 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.145 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.145 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.145 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.067938+0000", 2024-04-19T02:03:59.146 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.313e-06 2024-04-19T02:03:59.146 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.146 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.146 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.146 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.067990+0000", 2024-04-19T02:03:59.146 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2490000000000001e-05 2024-04-19T02:03:59.146 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.146 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.146 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.146 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.068014+0000", 2024-04-19T02:03:59.146 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4172000000000002e-05 2024-04-19T02:03:59.146 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.146 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.146 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.147 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.068034+0000", 2024-04-19T02:03:59.147 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9471999999999999e-05 2024-04-19T02:03:59.147 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.147 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.147 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.147 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.069268+0000", 2024-04-19T02:03:59.147 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001234165 2024-04-19T02:03:59.147 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.147 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.147 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.147 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.069293+0000", 2024-04-19T02:03:59.147 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4702000000000002e-05 2024-04-19T02:03:59.147 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.147 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.148 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.148 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.085653+0000", 2024-04-19T02:03:59.148 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.016360535999999998 2024-04-19T02:03:59.148 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.148 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.148 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.148 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.085743+0000", 2024-04-19T02:03:59.148 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.9264000000000005e-05 2024-04-19T02:03:59.148 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.148 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.148 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.148 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.148 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.148 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.1s3 46/26 ECSubWrite(tid=20, reqid=client.4509.0:415, at_version=46'8, trim_to=0'0, roll_forward_to=29'6))", 2024-04-19T02:03:59.148 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.077639+0000", 2024-04-19T02:03:59.149 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 29.018427207999999, 2024-04-19T02:03:59.149 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010858876999999999, 2024-04-19T02:03:59.149 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.149 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.149 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.149 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.149 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.149 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.149 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.077639+0000", 2024-04-19T02:03:59.149 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.149 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.149 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.149 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.149 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.077639+0000", 2024-04-19T02:03:59.149 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.150 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.150 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.150 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.150 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.077640+0000", 2024-04-19T02:03:59.150 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1999999999999999e-07 2024-04-19T02:03:59.150 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.150 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.150 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.150 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078039+0000", 2024-04-19T02:03:59.150 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00039909899999999998 2024-04-19T02:03:59.150 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.150 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.150 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.150 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078071+0000", 2024-04-19T02:03:59.151 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1987999999999999e-05 2024-04-19T02:03:59.151 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.151 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.151 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.151 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078093+0000", 2024-04-19T02:03:59.151 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2086000000000002e-05 2024-04-19T02:03:59.151 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.151 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.151 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.151 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078730+0000", 2024-04-19T02:03:59.151 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00063664699999999995 2024-04-19T02:03:59.151 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.151 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.151 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.152 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078755+0000", 2024-04-19T02:03:59.152 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5029e-05 2024-04-19T02:03:59.152 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.152 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.152 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.152 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.088431+0000", 2024-04-19T02:03:59.152 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0096761820000000002 2024-04-19T02:03:59.152 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.152 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.152 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.152 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.088498+0000", 2024-04-19T02:03:59.152 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7125999999999994e-05 2024-04-19T02:03:59.152 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.153 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.153 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.153 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.153 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.153 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.2s1 46/26 ECSubWrite(tid=159, reqid=osd.3.0:159, at_version=46'75, trim_to=0'0, roll_forward_to=45'73))", 2024-04-19T02:03:59.153 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.078187+0000", 2024-04-19T02:03:59.153 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 29.017879240999999, 2024-04-19T02:03:59.153 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0097959210000000008, 2024-04-19T02:03:59.153 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.153 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.153 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.153 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.153 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.154 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.154 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078187+0000", 2024-04-19T02:03:59.154 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.154 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.154 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.154 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.154 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078187+0000", 2024-04-19T02:03:59.154 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.154 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.154 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.154 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.154 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078188+0000", 2024-04-19T02:03:59.154 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.39e-06 2024-04-19T02:03:59.154 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.155 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.155 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.155 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078235+0000", 2024-04-19T02:03:59.155 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6297000000000001e-05 2024-04-19T02:03:59.155 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.155 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.155 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.155 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078262+0000", 2024-04-19T02:03:59.155 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7399e-05 2024-04-19T02:03:59.155 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.155 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.155 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.155 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078282+0000", 2024-04-19T02:03:59.155 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9908e-05 2024-04-19T02:03:59.156 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.156 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.156 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.156 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.079243+0000", 2024-04-19T02:03:59.156 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00096095200000000005 2024-04-19T02:03:59.156 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.156 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.156 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.156 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.079266+0000", 2024-04-19T02:03:59.156 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2994999999999999e-05 2024-04-19T02:03:59.156 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.156 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.156 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.156 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.087819+0000", 2024-04-19T02:03:59.156 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0085527599999999995 2024-04-19T02:03:59.157 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.157 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.157 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.157 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.087983+0000", 2024-04-19T02:03:59.157 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016422 2024-04-19T02:03:59.157 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.157 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.157 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.157 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.157 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.157 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.2s1 46/26 ECSubWrite(tid=163, reqid=client.4509.0:425, at_version=46'78, trim_to=0'0, roll_forward_to=46'75))", 2024-04-19T02:03:59.157 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.196489+0000", 2024-04-19T02:03:59.157 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 28.899577259000001, 2024-04-19T02:03:59.157 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013644882000000001, 2024-04-19T02:03:59.158 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.158 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.158 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.158 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.158 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.158 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.158 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196489+0000", 2024-04-19T02:03:59.158 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.158 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.158 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.158 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.158 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196489+0000", 2024-04-19T02:03:59.158 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.158 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.159 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.159 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.159 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196491+0000", 2024-04-19T02:03:59.159 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4269999999999998e-06 2024-04-19T02:03:59.159 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.159 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.159 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.159 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196543+0000", 2024-04-19T02:03:59.159 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1112999999999997e-05 2024-04-19T02:03:59.159 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.159 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.159 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.159 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196607+0000", 2024-04-19T02:03:59.159 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4183000000000005e-05 2024-04-19T02:03:59.159 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.160 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.160 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.160 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196634+0000", 2024-04-19T02:03:59.160 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6968e-05 2024-04-19T02:03:59.160 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.160 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.160 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.160 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.198568+0000", 2024-04-19T02:03:59.160 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0019338129999999999 2024-04-19T02:03:59.160 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.160 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.160 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.160 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.198591+0000", 2024-04-19T02:03:59.160 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3898000000000001e-05 2024-04-19T02:03:59.160 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.161 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.161 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.161 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.210049+0000", 2024-04-19T02:03:59.161 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011457355000000001 2024-04-19T02:03:59.161 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.161 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.161 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.161 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.210134+0000", 2024-04-19T02:03:59.161 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5124999999999996e-05 2024-04-19T02:03:59.161 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.161 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.161 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.161 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.161 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.161 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.ds4 46/26 ECSubWrite(tid=190, reqid=client.4509.0:428, at_version=46'37, trim_to=0'0, roll_forward_to=46'35))", 2024-04-19T02:03:59.162 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.196834+0000", 2024-04-19T02:03:59.162 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 28.899231975999999, 2024-04-19T02:03:59.162 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011503825000000001, 2024-04-19T02:03:59.162 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.162 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.162 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.162 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.162 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.162 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.162 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196834+0000", 2024-04-19T02:03:59.162 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.162 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.162 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.162 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.162 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196834+0000", 2024-04-19T02:03:59.163 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.163 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.163 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.163 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.163 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196835+0000", 2024-04-19T02:03:59.163 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.13e-06 2024-04-19T02:03:59.163 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.163 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.163 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.163 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196895+0000", 2024-04-19T02:03:59.163 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9481999999999997e-05 2024-04-19T02:03:59.163 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.163 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.163 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.163 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196922+0000", 2024-04-19T02:03:59.164 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6639e-05 2024-04-19T02:03:59.164 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.164 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.164 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.164 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196942+0000", 2024-04-19T02:03:59.164 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0931999999999999e-05 2024-04-19T02:03:59.164 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.164 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.164 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.164 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.197738+0000", 2024-04-19T02:03:59.164 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00079601499999999996 2024-04-19T02:03:59.164 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.164 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.164 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.164 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.197811+0000", 2024-04-19T02:03:59.165 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2916000000000005e-05 2024-04-19T02:03:59.165 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.165 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.165 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.165 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.208200+0000", 2024-04-19T02:03:59.165 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010388246 2024-04-19T02:03:59.165 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.165 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.165 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.165 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.208338+0000", 2024-04-19T02:03:59.165 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000138465 2024-04-19T02:03:59.165 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.165 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.165 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.166 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.166 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.166 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.2s1 46/26 ECSubWrite(tid=164, reqid=client.4509.0:426, at_version=46'79, trim_to=0'0, roll_forward_to=46'75))", 2024-04-19T02:03:59.166 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.199712+0000", 2024-04-19T02:03:59.166 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 28.896354571, 2024-04-19T02:03:59.166 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010489036, 2024-04-19T02:03:59.166 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.166 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.166 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.166 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.166 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.166 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.166 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.199712+0000", 2024-04-19T02:03:59.166 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.166 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.167 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.167 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.167 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.199712+0000", 2024-04-19T02:03:59.167 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.167 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.167 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.167 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.167 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.199713+0000", 2024-04-19T02:03:59.167 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7630000000000001e-06 2024-04-19T02:03:59.167 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.167 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.167 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.167 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.199758+0000", 2024-04-19T02:03:59.167 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4969999999999998e-05 2024-04-19T02:03:59.167 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.167 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.168 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.168 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.199784+0000", 2024-04-19T02:03:59.168 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5846e-05 2024-04-19T02:03:59.168 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.168 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.168 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.168 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.199805+0000", 2024-04-19T02:03:59.168 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0271999999999998e-05 2024-04-19T02:03:59.168 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.168 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.168 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.168 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.199970+0000", 2024-04-19T02:03:59.168 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016560899999999999 2024-04-19T02:03:59.168 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.169 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.169 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.169 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.199993+0000", 2024-04-19T02:03:59.169 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2549e-05 2024-04-19T02:03:59.169 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.169 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.169 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.169 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.210142+0000", 2024-04-19T02:03:59.169 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010149768 2024-04-19T02:03:59.169 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.169 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.169 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.169 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.210201+0000", 2024-04-19T02:03:59.169 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8258999999999999e-05 2024-04-19T02:03:59.169 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.170 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.170 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.170 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.170 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.170 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.1s3 47/26 ECSubWrite(tid=30, reqid=osd.0.0:30, at_version=47'17, trim_to=0'0, roll_forward_to=46'13))", 2024-04-19T02:03:59.170 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.085565+0000", 2024-04-19T02:03:59.170 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 28.010501157, 2024-04-19T02:03:59.170 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010262217000000001, 2024-04-19T02:03:59.170 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.170 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.170 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.170 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.170 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.170 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.170 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085565+0000", 2024-04-19T02:03:59.171 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.171 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.171 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.171 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.171 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085565+0000", 2024-04-19T02:03:59.171 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.171 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.171 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.171 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.171 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085566+0000", 2024-04-19T02:03:59.171 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.33e-07 2024-04-19T02:03:59.171 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.171 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.171 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.171 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085617+0000", 2024-04-19T02:03:59.171 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0487e-05 2024-04-19T02:03:59.172 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.172 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.172 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.172 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085639+0000", 2024-04-19T02:03:59.172 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2866000000000001e-05 2024-04-19T02:03:59.172 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.172 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.172 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.172 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085658+0000", 2024-04-19T02:03:59.172 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8575000000000001e-05 2024-04-19T02:03:59.172 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.172 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.172 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.172 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.090331+0000", 2024-04-19T02:03:59.172 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0046729650000000003 2024-04-19T02:03:59.172 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.173 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.173 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.173 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.090358+0000", 2024-04-19T02:03:59.173 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7424999999999999e-05 2024-04-19T02:03:59.173 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.173 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.173 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.173 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.095729+0000", 2024-04-19T02:03:59.173 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0053710989999999998 2024-04-19T02:03:59.173 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.173 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.173 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.173 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.095827+0000", 2024-04-19T02:03:59.173 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7867000000000003e-05 2024-04-19T02:03:59.173 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.174 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.174 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.174 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.174 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.174 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.8s4 66/26 ECSubWrite(tid=150, reqid=client.4509.0:722, at_version=66'65, trim_to=0'0, roll_forward_to=57'63))", 2024-04-19T02:03:59.174 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:50.060860+0000", 2024-04-19T02:03:59.174 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 9.0352065079999999, 2024-04-19T02:03:59.174 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014413267, 2024-04-19T02:03:59.174 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.174 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.174 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.174 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.174 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.174 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.175 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.060860+0000", 2024-04-19T02:03:59.175 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.175 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.175 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.175 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.175 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.060860+0000", 2024-04-19T02:03:59.175 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.175 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.175 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.175 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.175 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.060860+0000", 2024-04-19T02:03:59.175 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0299999999999999e-07 2024-04-19T02:03:59.175 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.175 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.175 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.176 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.060914+0000", 2024-04-19T02:03:59.176 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3627e-05 2024-04-19T02:03:59.176 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.176 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.176 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.176 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.060943+0000", 2024-04-19T02:03:59.176 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9461000000000001e-05 2024-04-19T02:03:59.176 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.176 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.176 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.176 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.060963+0000", 2024-04-19T02:03:59.176 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9168e-05 2024-04-19T02:03:59.176 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.176 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.176 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.177 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.061169+0000", 2024-04-19T02:03:59.177 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00020598 2024-04-19T02:03:59.177 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.177 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.177 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.177 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.061196+0000", 2024-04-19T02:03:59.177 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7631999999999999e-05 2024-04-19T02:03:59.177 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.177 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.177 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.177 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.075230+0000", 2024-04-19T02:03:59.177 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014033990999999999 2024-04-19T02:03:59.177 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.177 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.178 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.178 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.075273+0000", 2024-04-19T02:03:59.178 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2805000000000001e-05 2024-04-19T02:03:59.178 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.178 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.178 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.178 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.178 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.178 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.8s4 66/26 ECSubWrite(tid=151, reqid=client.4509.0:723, at_version=66'66, trim_to=0'0, roll_forward_to=57'63))", 2024-04-19T02:03:59.178 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:50.065768+0000", 2024-04-19T02:03:59.178 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 9.0302987120000004, 2024-04-19T02:03:59.178 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0095560639999999995, 2024-04-19T02:03:59.178 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.179 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.179 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.179 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.179 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.179 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.179 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.065768+0000", 2024-04-19T02:03:59.179 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.179 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.179 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.179 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.179 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.065768+0000", 2024-04-19T02:03:59.179 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.179 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.179 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.180 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.180 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.065770+0000", 2024-04-19T02:03:59.180 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.074e-06 2024-04-19T02:03:59.180 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.180 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.180 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.180 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.065820+0000", 2024-04-19T02:03:59.180 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9874999999999999e-05 2024-04-19T02:03:59.180 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.180 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.180 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.180 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.065853+0000", 2024-04-19T02:03:59.180 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3257999999999999e-05 2024-04-19T02:03:59.180 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.180 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.181 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.181 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.065876+0000", 2024-04-19T02:03:59.181 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3186000000000001e-05 2024-04-19T02:03:59.181 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.181 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.181 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.181 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.066119+0000", 2024-04-19T02:03:59.181 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000242678 2024-04-19T02:03:59.181 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.181 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.181 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.181 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.066145+0000", 2024-04-19T02:03:59.181 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6359e-05 2024-04-19T02:03:59.181 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.182 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.182 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.182 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.075290+0000", 2024-04-19T02:03:59.182 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0091445160000000001 2024-04-19T02:03:59.182 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.182 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.182 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.182 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.075324+0000", 2024-04-19T02:03:59.182 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4118000000000001e-05 2024-04-19T02:03:59.182 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.182 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.182 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.182 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.182 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.183 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:59.185 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-19T02:03:59.185 INFO:tasks.rados.rados.0.smithi059.stdout:318: done (5 left) 2024-04-19T02:03:59.186 INFO:tasks.rados.rados.0.smithi059.stdout:320: done (4 left) 2024-04-19T02:03:59.186 INFO:tasks.rados.rados.0.smithi059.stdout:321: done (3 left) 2024-04-19T02:03:59.186 INFO:tasks.rados.rados.0.smithi059.stdout:322: done (2 left) 2024-04-19T02:03:59.186 INFO:tasks.rados.rados.0.smithi059.stdout:323: done (1 left) 2024-04-19T02:03:59.186 INFO:tasks.rados.rados.0.smithi059.stdout:324: done (0 left) 2024-04-19T02:03:59.186 INFO:tasks.rados.rados.0.smithi059.stdout:325: rollback oid 29 current snap is 27 2024-04-19T02:03:59.186 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 29 to 23 2024-04-19T02:03:59.186 INFO:tasks.rados.rados.0.smithi059.stdout:326: read oid 24 snap -1 2024-04-19T02:03:59.186 INFO:tasks.rados.rados.0.smithi059.stdout:326: expect deleted 2024-04-19T02:03:59.186 INFO:tasks.rados.rados.0.smithi059.stdout:327: rollback oid 31 current snap is 27 2024-04-19T02:03:59.186 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 31 to 21 2024-04-19T02:03:59.186 INFO:tasks.rados.rados.0.smithi059.stdout:328: snap_create 2024-04-19T02:03:59.186 INFO:tasks.rados.rados.0.smithi059.stdout:327: finishing rollback tid 0 to smithi05930959-31 2024-04-19T02:03:59.187 INFO:tasks.rados.rados.0.smithi059.stdout:327: finishing rollback tid 1 to smithi05930959-31 2024-04-19T02:03:59.187 INFO:tasks.rados.rados.0.smithi059.stdout:327: finishing rollback tid 2 to smithi05930959-31 2024-04-19T02:03:59.187 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 31 v 95 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-19T02:03:59.187 INFO:tasks.rados.rados.0.smithi059.stdout:325: finishing rollback tid 0 to smithi05930959-29 2024-04-19T02:03:59.187 INFO:tasks.rados.rados.0.smithi059.stdout:325: finishing rollback tid 1 to smithi05930959-29 2024-04-19T02:03:59.187 INFO:tasks.rados.rados.0.smithi059.stdout:325: finishing rollback tid 2 to smithi05930959-29 2024-04-19T02:03:59.187 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 29 v 51 (ObjNum 138 snap 22 seq_num 138) dirty exists 2024-04-19T02:03:59.188 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:03:59.185+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:03:59.285 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:03:59.285+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:03:59.309 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:59.309 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:59.309 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 0 2024-04-19T02:03:59.309 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:59.323 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-19T02:03:59.411 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:03:59.413+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:03:59.487 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:59.485+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:03:59.491 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:59.491 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:59.491 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-04-19T02:03:59.491 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-04-19T02:03:59.491 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:59.505 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-19T02:03:59.588 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:59.589+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:59.672 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:59.672 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-04-19T02:03:59.672 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-04-19T02:03:59.672 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-04-19T02:03:59.672 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.673 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.8s6 42/26 ECSubWrite(tid=71, reqid=osd.4.0:71, at_version=42'33, trim_to=0'0, roll_forward_to=37'31))", 2024-04-19T02:03:59.673 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.746437+0000", 2024-04-19T02:03:59.673 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 32.925567264999998, 2024-04-19T02:03:59.673 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011548248000000001, 2024-04-19T02:03:59.673 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.673 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.673 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.673 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.673 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.673 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.673 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746437+0000", 2024-04-19T02:03:59.673 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.673 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.673 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.673 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.674 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746437+0000", 2024-04-19T02:03:59.674 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.674 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.674 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.674 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.674 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746438+0000", 2024-04-19T02:03:59.674 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6000000000000003e-07 2024-04-19T02:03:59.674 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.674 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.674 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.674 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746491+0000", 2024-04-19T02:03:59.674 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2939000000000003e-05 2024-04-19T02:03:59.674 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.674 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.675 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.675 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746520+0000", 2024-04-19T02:03:59.675 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8835000000000001e-05 2024-04-19T02:03:59.675 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.675 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.675 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.675 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746541+0000", 2024-04-19T02:03:59.675 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1628999999999999e-05 2024-04-19T02:03:59.675 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.675 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.675 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.675 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746712+0000", 2024-04-19T02:03:59.675 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017087100000000001 2024-04-19T02:03:59.675 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.675 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.676 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.676 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746742+0000", 2024-04-19T02:03:59.676 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9691000000000001e-05 2024-04-19T02:03:59.676 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.676 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.676 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.676 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.757878+0000", 2024-04-19T02:03:59.676 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011136546000000001 2024-04-19T02:03:59.676 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.676 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.676 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.676 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.757985+0000", 2024-04-19T02:03:59.676 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000106977 2024-04-19T02:03:59.676 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.677 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.677 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.677 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.677 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.677 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.bs3 42/26 ECSubWrite(tid=133, reqid=osd.3.0:133, at_version=42'21, trim_to=0'0, roll_forward_to=39'19))", 2024-04-19T02:03:59.677 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.748283+0000", 2024-04-19T02:03:59.677 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 32.923721807, 2024-04-19T02:03:59.677 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012079138999999999, 2024-04-19T02:03:59.677 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.677 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.677 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.677 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.677 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.677 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.677 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748283+0000", 2024-04-19T02:03:59.678 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.678 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.678 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.678 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.678 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748283+0000", 2024-04-19T02:03:59.678 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.678 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.678 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.678 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.678 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748283+0000", 2024-04-19T02:03:59.678 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3300000000000002e-07 2024-04-19T02:03:59.678 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.678 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.678 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.678 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748372+0000", 2024-04-19T02:03:59.679 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8783999999999996e-05 2024-04-19T02:03:59.679 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.679 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.679 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.679 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748403+0000", 2024-04-19T02:03:59.679 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1228e-05 2024-04-19T02:03:59.679 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.679 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.679 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.679 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748427+0000", 2024-04-19T02:03:59.679 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3512000000000001e-05 2024-04-19T02:03:59.679 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.679 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.679 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.679 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748811+0000", 2024-04-19T02:03:59.680 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00038409700000000002 2024-04-19T02:03:59.680 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.680 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.680 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.680 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748836+0000", 2024-04-19T02:03:59.680 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5364999999999999e-05 2024-04-19T02:03:59.680 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.680 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.680 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.680 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.760267+0000", 2024-04-19T02:03:59.680 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011430656000000001 2024-04-19T02:03:59.680 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.680 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.680 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.681 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.760362+0000", 2024-04-19T02:03:59.681 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4864000000000005e-05 2024-04-19T02:03:59.681 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.681 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.681 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.681 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.681 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.681 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.bs3 42/26 ECSubWrite(tid=135, reqid=osd.3.0:135, at_version=42'23, trim_to=0'0, roll_forward_to=39'19))", 2024-04-19T02:03:59.681 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.748483+0000", 2024-04-19T02:03:59.681 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 32.923520977000003, 2024-04-19T02:03:59.681 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.015459633, 2024-04-19T02:03:59.681 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.681 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.681 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.681 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.681 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.682 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.682 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748483+0000", 2024-04-19T02:03:59.682 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.682 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.682 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.682 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.682 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748483+0000", 2024-04-19T02:03:59.682 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.682 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.682 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.682 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.682 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748484+0000", 2024-04-19T02:03:59.682 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8899999999999998e-07 2024-04-19T02:03:59.683 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.683 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.683 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.683 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748527+0000", 2024-04-19T02:03:59.683 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2651000000000002e-05 2024-04-19T02:03:59.683 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.683 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.683 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.683 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748550+0000", 2024-04-19T02:03:59.683 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3345000000000001e-05 2024-04-19T02:03:59.683 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.683 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.683 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.683 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748569+0000", 2024-04-19T02:03:59.683 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8722000000000001e-05 2024-04-19T02:03:59.683 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.684 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.684 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.684 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.752605+0000", 2024-04-19T02:03:59.684 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0040362510000000002 2024-04-19T02:03:59.684 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.684 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.684 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.684 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.752629+0000", 2024-04-19T02:03:59.684 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3685000000000001e-05 2024-04-19T02:03:59.684 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.684 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.684 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.684 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.763825+0000", 2024-04-19T02:03:59.684 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011195577999999999 2024-04-19T02:03:59.685 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.685 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.685 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.685 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.763943+0000", 2024-04-19T02:03:59.685 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000118412 2024-04-19T02:03:59.685 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.685 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.685 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.685 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.685 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.685 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.5s6 42/26 ECSubWrite(tid=51, reqid=osd.10.0:51, at_version=42'27, trim_to=0'0, roll_forward_to=39'25))", 2024-04-19T02:03:59.685 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.749097+0000", 2024-04-19T02:03:59.685 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 32.922907002000002, 2024-04-19T02:03:59.685 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011834341, 2024-04-19T02:03:59.685 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.686 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.686 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.686 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.686 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.686 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.686 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749097+0000", 2024-04-19T02:03:59.686 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.686 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.686 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.686 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.686 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749097+0000", 2024-04-19T02:03:59.686 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.686 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.686 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.686 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.687 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749098+0000", 2024-04-19T02:03:59.687 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9299999999999997e-07 2024-04-19T02:03:59.687 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.687 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.687 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.687 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749170+0000", 2024-04-19T02:03:59.687 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2208999999999999e-05 2024-04-19T02:03:59.687 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.687 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.687 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.687 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749270+0000", 2024-04-19T02:03:59.687 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000100089 2024-04-19T02:03:59.687 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.687 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.688 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.688 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749297+0000", 2024-04-19T02:03:59.688 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6485000000000001e-05 2024-04-19T02:03:59.688 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.688 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.688 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.688 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749537+0000", 2024-04-19T02:03:59.688 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00024036700000000001 2024-04-19T02:03:59.688 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.688 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.688 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.688 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749562+0000", 2024-04-19T02:03:59.688 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5082000000000001e-05 2024-04-19T02:03:59.688 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.688 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.688 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.689 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.760847+0000", 2024-04-19T02:03:59.689 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011285165999999999 2024-04-19T02:03:59.689 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.689 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.689 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.689 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.760932+0000", 2024-04-19T02:03:59.689 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.4250000000000001e-05 2024-04-19T02:03:59.689 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.689 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.689 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.689 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.690 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.690 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.5s6 42/26 ECSubWrite(tid=52, reqid=osd.10.0:52, at_version=42'29, trim_to=0'0, roll_forward_to=39'25))", 2024-04-19T02:03:59.690 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.758688+0000", 2024-04-19T02:03:59.690 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 32.913315871999998, 2024-04-19T02:03:59.690 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0063222319999999997, 2024-04-19T02:03:59.690 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.690 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.690 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.690 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.690 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.690 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.690 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.758688+0000", 2024-04-19T02:03:59.690 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.690 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.691 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.691 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.691 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.758688+0000", 2024-04-19T02:03:59.691 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.691 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.691 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.691 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.691 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.758690+0000", 2024-04-19T02:03:59.691 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.81e-06 2024-04-19T02:03:59.691 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.691 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.691 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.691 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.758767+0000", 2024-04-19T02:03:59.691 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7218000000000005e-05 2024-04-19T02:03:59.691 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.692 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.692 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.692 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.758796+0000", 2024-04-19T02:03:59.692 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8538e-05 2024-04-19T02:03:59.692 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.692 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.692 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.692 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.758817+0000", 2024-04-19T02:03:59.692 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1404999999999999e-05 2024-04-19T02:03:59.692 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.692 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.692 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.692 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.758982+0000", 2024-04-19T02:03:59.692 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016446899999999999 2024-04-19T02:03:59.692 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.693 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.693 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.693 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.759014+0000", 2024-04-19T02:03:59.693 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.2407999999999999e-05 2024-04-19T02:03:59.693 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.693 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.693 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.693 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.764947+0000", 2024-04-19T02:03:59.694 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.005932522 2024-04-19T02:03:59.694 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.694 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.694 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.695 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.765011+0000", 2024-04-19T02:03:59.695 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3862000000000006e-05 2024-04-19T02:03:59.695 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.695 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.695 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.695 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.695 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.695 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.8s6 46/26 ECSubWrite(tid=81, reqid=osd.4.0:81, at_version=46'42, trim_to=0'0, roll_forward_to=45'40))", 2024-04-19T02:03:59.695 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.061957+0000", 2024-04-19T02:03:59.695 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 29.610047755, 2024-04-19T02:03:59.695 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0094780439999999997, 2024-04-19T02:03:59.695 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.695 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.695 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.695 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.696 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.696 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.696 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.061957+0000", 2024-04-19T02:03:59.696 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.696 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.696 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.696 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.696 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.061957+0000", 2024-04-19T02:03:59.696 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.696 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.696 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.696 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.696 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.061957+0000", 2024-04-19T02:03:59.696 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.4700000000000001e-07 2024-04-19T02:03:59.696 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.697 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.697 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.697 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.062014+0000", 2024-04-19T02:03:59.697 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6895999999999999e-05 2024-04-19T02:03:59.697 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.697 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.697 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.697 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.062048+0000", 2024-04-19T02:03:59.697 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3597999999999999e-05 2024-04-19T02:03:59.697 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.697 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.697 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.697 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.062072+0000", 2024-04-19T02:03:59.697 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4204999999999999e-05 2024-04-19T02:03:59.697 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.698 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.698 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.698 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.062301+0000", 2024-04-19T02:03:59.698 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00022887999999999999 2024-04-19T02:03:59.698 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.698 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.698 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.698 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.062328+0000", 2024-04-19T02:03:59.698 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7280999999999999e-05 2024-04-19T02:03:59.698 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.698 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.698 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.698 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.071333+0000", 2024-04-19T02:03:59.698 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0090052770000000008 2024-04-19T02:03:59.699 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.699 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.699 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.699 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.071435+0000", 2024-04-19T02:03:59.699 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010116 2024-04-19T02:03:59.699 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.699 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.699 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.699 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.699 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.699 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.7s1 46/26 ECSubWrite(tid=158, reqid=osd.3.0:158, at_version=46'14, trim_to=0'0, roll_forward_to=45'12))", 2024-04-19T02:03:59.699 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.072342+0000", 2024-04-19T02:03:59.699 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 29.599662512999998, 2024-04-19T02:03:59.699 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0063685260000000002, 2024-04-19T02:03:59.700 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.700 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.700 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.700 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.700 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.700 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.700 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072342+0000", 2024-04-19T02:03:59.700 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.700 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.700 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.700 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.700 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072342+0000", 2024-04-19T02:03:59.700 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.700 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.701 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.701 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.701 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072343+0000", 2024-04-19T02:03:59.701 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9999999999999997e-07 2024-04-19T02:03:59.701 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.701 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.701 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.701 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072389+0000", 2024-04-19T02:03:59.701 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6427000000000001e-05 2024-04-19T02:03:59.701 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.701 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.701 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.701 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072416+0000", 2024-04-19T02:03:59.701 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7209e-05 2024-04-19T02:03:59.701 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.702 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.702 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.702 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072437+0000", 2024-04-19T02:03:59.702 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0874999999999999e-05 2024-04-19T02:03:59.702 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.702 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.702 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.702 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072649+0000", 2024-04-19T02:03:59.702 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000211821 2024-04-19T02:03:59.702 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.702 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.702 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.702 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072674+0000", 2024-04-19T02:03:59.702 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4734000000000001e-05 2024-04-19T02:03:59.703 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.703 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.703 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.703 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078610+0000", 2024-04-19T02:03:59.703 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0059367899999999999 2024-04-19T02:03:59.703 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.703 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.703 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.703 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078710+0000", 2024-04-19T02:03:59.703 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9970000000000007e-05 2024-04-19T02:03:59.703 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.703 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.704 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.704 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.704 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.704 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.bs3 46/26 ECSubWrite(tid=177, reqid=client.4509.0:437, at_version=46'34, trim_to=0'0, roll_forward_to=46'32))", 2024-04-19T02:03:59.704 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.239790+0000", 2024-04-19T02:03:59.704 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 29.432214437999999, 2024-04-19T02:03:59.704 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0049614730000000001, 2024-04-19T02:03:59.704 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.704 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.704 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.704 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.704 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.704 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.704 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.239790+0000", 2024-04-19T02:03:59.705 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.705 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.705 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.705 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.705 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.239790+0000", 2024-04-19T02:03:59.705 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.705 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.705 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.705 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.705 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.239792+0000", 2024-04-19T02:03:59.705 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1730000000000002e-06 2024-04-19T02:03:59.705 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.705 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.705 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.705 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.240853+0000", 2024-04-19T02:03:59.705 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001060725 2024-04-19T02:03:59.706 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.706 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.706 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.706 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.240899+0000", 2024-04-19T02:03:59.706 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.5933999999999998e-05 2024-04-19T02:03:59.706 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.706 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.706 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.706 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.240923+0000", 2024-04-19T02:03:59.706 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4426e-05 2024-04-19T02:03:59.706 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.706 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.706 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.706 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.241100+0000", 2024-04-19T02:03:59.707 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017696000000000001 2024-04-19T02:03:59.707 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.707 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.707 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.707 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.241127+0000", 2024-04-19T02:03:59.707 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7180999999999999e-05 2024-04-19T02:03:59.707 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.707 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.707 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.707 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.244633+0000", 2024-04-19T02:03:59.707 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0035052410000000001 2024-04-19T02:03:59.707 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.707 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.707 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.707 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.244751+0000", 2024-04-19T02:03:59.708 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011883300000000001 2024-04-19T02:03:59.708 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.708 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.708 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.708 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.708 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.708 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.bs3 47/26 ECSubWrite(tid=183, reqid=osd.3.0:183, at_version=47'38, trim_to=0'0, roll_forward_to=46'34))", 2024-04-19T02:03:59.708 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.081643+0000", 2024-04-19T02:03:59.708 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 28.590360929999999, 2024-04-19T02:03:59.708 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0080822910000000001, 2024-04-19T02:03:59.708 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.708 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.708 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.708 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.708 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.709 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.709 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081643+0000", 2024-04-19T02:03:59.709 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.709 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.709 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.709 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.709 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081643+0000", 2024-04-19T02:03:59.709 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.709 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.709 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.709 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.709 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081644+0000", 2024-04-19T02:03:59.709 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8700000000000005e-07 2024-04-19T02:03:59.709 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.709 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.710 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.710 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081684+0000", 2024-04-19T02:03:59.710 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0271000000000001e-05 2024-04-19T02:03:59.710 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.710 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.710 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.710 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081706+0000", 2024-04-19T02:03:59.710 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1622000000000001e-05 2024-04-19T02:03:59.710 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.710 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.710 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.710 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.081722+0000", 2024-04-19T02:03:59.710 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6322999999999999e-05 2024-04-19T02:03:59.710 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.710 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.711 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.711 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.083858+0000", 2024-04-19T02:03:59.711 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0021358979999999998 2024-04-19T02:03:59.711 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.711 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.711 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.711 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.083891+0000", 2024-04-19T02:03:59.711 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.2604999999999997e-05 2024-04-19T02:03:59.711 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.711 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.711 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.711 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.089628+0000", 2024-04-19T02:03:59.711 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0057371139999999998 2024-04-19T02:03:59.711 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.711 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.712 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.712 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.089726+0000", 2024-04-19T02:03:59.712 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7670999999999994e-05 2024-04-19T02:03:59.712 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.712 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.712 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.712 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.712 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.712 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.8s6 47/26 ECSubWrite(tid=86, reqid=osd.4.0:86, at_version=47'44, trim_to=0'0, roll_forward_to=46'42))", 2024-04-19T02:03:59.712 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.091163+0000", 2024-04-19T02:03:59.712 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 28.580841554999999, 2024-04-19T02:03:59.712 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.005003281, 2024-04-19T02:03:59.712 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.712 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.712 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.713 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.713 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.713 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.713 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.091163+0000", 2024-04-19T02:03:59.713 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.713 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.713 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.713 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.713 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.091163+0000", 2024-04-19T02:03:59.713 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.713 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.713 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.713 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.713 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.091163+0000", 2024-04-19T02:03:59.713 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5700000000000002e-07 2024-04-19T02:03:59.714 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.714 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.714 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.714 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.091212+0000", 2024-04-19T02:03:59.714 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.8269999999999997e-05 2024-04-19T02:03:59.714 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.714 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.714 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.714 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.091252+0000", 2024-04-19T02:03:59.714 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0515000000000001e-05 2024-04-19T02:03:59.714 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.714 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.714 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.714 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.091274+0000", 2024-04-19T02:03:59.715 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2130999999999999e-05 2024-04-19T02:03:59.715 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.715 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.715 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.715 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.091476+0000", 2024-04-19T02:03:59.715 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00020131200000000001 2024-04-19T02:03:59.715 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.715 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.715 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.715 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.091500+0000", 2024-04-19T02:03:59.715 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4771999999999999e-05 2024-04-19T02:03:59.715 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.715 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.715 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.716 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.096029+0000", 2024-04-19T02:03:59.716 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0045288459999999996 2024-04-19T02:03:59.716 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.716 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.716 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.716 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.096166+0000", 2024-04-19T02:03:59.716 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000136778 2024-04-19T02:03:59.716 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.716 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.716 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.716 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.716 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.716 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.8s6 47/26 ECSubWrite(tid=87, reqid=osd.4.0:86, at_version=0'0, trim_to=0'0, roll_forward_to=47'44))", 2024-04-19T02:03:59.716 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.118646+0000", 2024-04-19T02:03:59.717 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 28.553357862999999, 2024-04-19T02:03:59.717 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0060891230000000001, 2024-04-19T02:03:59.717 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.717 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.717 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.717 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.717 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.717 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.717 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.118646+0000", 2024-04-19T02:03:59.717 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.717 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.717 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.717 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.718 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.118646+0000", 2024-04-19T02:03:59.718 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.718 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.718 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.718 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.718 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.118648+0000", 2024-04-19T02:03:59.718 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.843e-06 2024-04-19T02:03:59.718 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.718 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.718 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.718 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.118709+0000", 2024-04-19T02:03:59.718 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.1147000000000003e-05 2024-04-19T02:03:59.718 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.718 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.719 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.719 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.118752+0000", 2024-04-19T02:03:59.719 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2997000000000002e-05 2024-04-19T02:03:59.719 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.719 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.719 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.719 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.118776+0000", 2024-04-19T02:03:59.719 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3189000000000001e-05 2024-04-19T02:03:59.719 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.719 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.719 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.719 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.119269+0000", 2024-04-19T02:03:59.720 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00049325300000000003 2024-04-19T02:03:59.720 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.720 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.720 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.720 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.119294+0000", 2024-04-19T02:03:59.720 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5381999999999999e-05 2024-04-19T02:03:59.720 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.720 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.720 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.720 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.124635+0000", 2024-04-19T02:03:59.720 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0053407120000000001 2024-04-19T02:03:59.720 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.720 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.721 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.721 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.124736+0000", 2024-04-19T02:03:59.721 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010060000000000001 2024-04-19T02:03:59.721 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.721 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.721 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.721 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.721 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.721 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.bs3 47/26 ECSubWrite(tid=186, reqid=osd.3.0:183, at_version=0'0, trim_to=0'0, roll_forward_to=47'38))", 2024-04-19T02:03:59.721 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.120984+0000", 2024-04-19T02:03:59.721 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 28.551020831999999, 2024-04-19T02:03:59.721 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0064153229999999997, 2024-04-19T02:03:59.721 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.721 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.721 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.722 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.722 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.722 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.722 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.120984+0000", 2024-04-19T02:03:59.722 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.722 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.722 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.722 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.722 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.120984+0000", 2024-04-19T02:03:59.722 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.722 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.722 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.722 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.722 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.120985+0000", 2024-04-19T02:03:59.723 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9460000000000001e-06 2024-04-19T02:03:59.723 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.723 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.723 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.723 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.121014+0000", 2024-04-19T02:03:59.723 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8648000000000001e-05 2024-04-19T02:03:59.723 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.723 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.723 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.723 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.121049+0000", 2024-04-19T02:03:59.723 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4912e-05 2024-04-19T02:03:59.723 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.723 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.723 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.723 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.121072+0000", 2024-04-19T02:03:59.724 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3095000000000002e-05 2024-04-19T02:03:59.724 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.724 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.724 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.724 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.121286+0000", 2024-04-19T02:03:59.724 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00021347400000000001 2024-04-19T02:03:59.724 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.724 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.724 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.724 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.121311+0000", 2024-04-19T02:03:59.724 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5820999999999999e-05 2024-04-19T02:03:59.724 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.724 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.724 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.724 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.127303+0000", 2024-04-19T02:03:59.725 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0059911920000000002 2024-04-19T02:03:59.725 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.725 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.725 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.725 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.127399+0000", 2024-04-19T02:03:59.725 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.6235000000000003e-05 2024-04-19T02:03:59.725 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.725 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.725 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.725 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.725 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.725 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.8s6 56/26 ECSubWrite(tid=110, reqid=client.4509.0:576, at_version=56'57, trim_to=0'0, roll_forward_to=55'55))", 2024-04-19T02:03:59.725 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:40.168846+0000", 2024-04-19T02:03:59.725 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 19.503158813999999, 2024-04-19T02:03:59.725 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0055459899999999998, 2024-04-19T02:03:59.725 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.726 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.726 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.726 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.726 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.726 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.726 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.168846+0000", 2024-04-19T02:03:59.726 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.726 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.726 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.726 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.726 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.168846+0000", 2024-04-19T02:03:59.726 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.726 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.726 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.726 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.727 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.168846+0000", 2024-04-19T02:03:59.727 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.7e-07 2024-04-19T02:03:59.727 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.727 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.727 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.727 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.169024+0000", 2024-04-19T02:03:59.727 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017762 2024-04-19T02:03:59.727 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.727 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.727 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.727 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.169049+0000", 2024-04-19T02:03:59.727 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5239e-05 2024-04-19T02:03:59.727 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.727 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.728 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.728 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.169061+0000", 2024-04-19T02:03:59.728 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2187e-05 2024-04-19T02:03:59.728 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.728 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.728 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.728 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.169903+0000", 2024-04-19T02:03:59.728 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00084208099999999995 2024-04-19T02:03:59.728 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.728 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.728 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.728 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.169929+0000", 2024-04-19T02:03:59.728 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6086000000000001e-05 2024-04-19T02:03:59.729 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.729 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.729 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.729 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.174340+0000", 2024-04-19T02:03:59.729 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0044111869999999996 2024-04-19T02:03:59.729 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.729 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.729 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.729 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.174392+0000", 2024-04-19T02:03:59.729 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1220000000000001e-05 2024-04-19T02:03:59.729 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.729 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.729 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.730 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.730 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.730 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.8s6 57/26 ECSubWrite(tid=120, reqid=osd.4.0:120, at_version=57'63, trim_to=0'0, roll_forward_to=56'61))", 2024-04-19T02:03:59.730 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:41.150828+0000", 2024-04-19T02:03:59.730 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 18.521175898999999, 2024-04-19T02:03:59.730 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0083979850000000002, 2024-04-19T02:03:59.730 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.730 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.730 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.730 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.730 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.730 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.730 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:41.150828+0000", 2024-04-19T02:03:59.730 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.731 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.731 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.731 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.731 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:41.150828+0000", 2024-04-19T02:03:59.731 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.731 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.731 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.731 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.731 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:41.150829+0000", 2024-04-19T02:03:59.731 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8599999999999999e-07 2024-04-19T02:03:59.731 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.731 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.731 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.732 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:41.150869+0000", 2024-04-19T02:03:59.732 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9990000000000002e-05 2024-04-19T02:03:59.732 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.732 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.732 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.732 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:41.150898+0000", 2024-04-19T02:03:59.732 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9164000000000001e-05 2024-04-19T02:03:59.732 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.732 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.732 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.732 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:41.150908+0000", 2024-04-19T02:03:59.732 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0512e-05 2024-04-19T02:03:59.732 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.732 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.732 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.733 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:41.151047+0000", 2024-04-19T02:03:59.733 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000138236 2024-04-19T02:03:59.733 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.733 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.733 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.733 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:41.151073+0000", 2024-04-19T02:03:59.733 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6372000000000001e-05 2024-04-19T02:03:59.733 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.733 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.733 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.733 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:41.159136+0000", 2024-04-19T02:03:59.733 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.008062975 2024-04-19T02:03:59.733 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.733 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.734 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.734 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:41.159226+0000", 2024-04-19T02:03:59.734 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0450000000000003e-05 2024-04-19T02:03:59.734 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.734 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.734 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.734 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.734 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.734 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.3s7 57/26 ECSubWrite(tid=76, reqid=osd.10.0:76, at_version=57'24, trim_to=0'0, roll_forward_to=56'22))", 2024-04-19T02:03:59.734 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:41.160369+0000", 2024-04-19T02:03:59.734 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 18.511635505000001, 2024-04-19T02:03:59.734 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0076937409999999996, 2024-04-19T02:03:59.734 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.734 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.735 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.735 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.735 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.735 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.735 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:41.160369+0000", 2024-04-19T02:03:59.735 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.735 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.735 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.735 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.735 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:41.160369+0000", 2024-04-19T02:03:59.735 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.735 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.735 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.735 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.735 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:41.160369+0000", 2024-04-19T02:03:59.736 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8400000000000004e-07 2024-04-19T02:03:59.736 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.736 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.736 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.736 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:41.160391+0000", 2024-04-19T02:03:59.736 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1508000000000001e-05 2024-04-19T02:03:59.736 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.736 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.736 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.736 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:41.160407+0000", 2024-04-19T02:03:59.736 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6104999999999999e-05 2024-04-19T02:03:59.736 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.736 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.736 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.736 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:41.160418+0000", 2024-04-19T02:03:59.737 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0746e-05 2024-04-19T02:03:59.737 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.737 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.737 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.737 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:41.160546+0000", 2024-04-19T02:03:59.737 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012792599999999999 2024-04-19T02:03:59.737 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.737 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.737 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.737 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:41.160570+0000", 2024-04-19T02:03:59.737 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4290999999999998e-05 2024-04-19T02:03:59.737 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.737 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.737 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.737 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:41.167975+0000", 2024-04-19T02:03:59.738 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0074052049999999998 2024-04-19T02:03:59.738 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.738 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.738 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.738 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:41.168063+0000", 2024-04-19T02:03:59.738 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7375999999999998e-05 2024-04-19T02:03:59.738 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.738 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.738 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.738 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.738 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.738 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.7s1 57/26 ECSubWrite(tid=228, reqid=osd.3.0:228, at_version=57'20, trim_to=0'0, roll_forward_to=56'18))", 2024-04-19T02:03:59.738 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:41.161133+0000", 2024-04-19T02:03:59.738 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 18.510870849, 2024-04-19T02:03:59.738 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0090808880000000005, 2024-04-19T02:03:59.739 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.739 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.739 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.739 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.739 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.739 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.739 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:41.161133+0000", 2024-04-19T02:03:59.739 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.739 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.739 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.739 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.739 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:41.161133+0000", 2024-04-19T02:03:59.739 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.739 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.739 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.739 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.740 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:41.161134+0000", 2024-04-19T02:03:59.740 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7400000000000002e-07 2024-04-19T02:03:59.740 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.740 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.740 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.740 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:41.161186+0000", 2024-04-19T02:03:59.740 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1551000000000002e-05 2024-04-19T02:03:59.740 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.740 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.740 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.740 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:41.161215+0000", 2024-04-19T02:03:59.740 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9065e-05 2024-04-19T02:03:59.740 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.740 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.740 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.741 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:41.161237+0000", 2024-04-19T02:03:59.741 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1685999999999999e-05 2024-04-19T02:03:59.741 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.741 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.741 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.741 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:41.161451+0000", 2024-04-19T02:03:59.741 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00021434700000000001 2024-04-19T02:03:59.741 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.741 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.741 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.741 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:41.161476+0000", 2024-04-19T02:03:59.741 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4655e-05 2024-04-19T02:03:59.741 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.741 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.741 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.741 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:41.170172+0000", 2024-04-19T02:03:59.742 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0086968140000000006 2024-04-19T02:03:59.742 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.742 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.742 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.742 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:41.170214+0000", 2024-04-19T02:03:59.742 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1996000000000002e-05 2024-04-19T02:03:59.742 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.742 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.742 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.742 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.742 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.742 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.bs3 62/26 ECSubWrite(tid=235, reqid=client.4509.0:649, at_version=62'40, trim_to=0'0, roll_forward_to=47'38))", 2024-04-19T02:03:59.742 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:46.214412+0000", 2024-04-19T02:03:59.743 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 13.457592663, 2024-04-19T02:03:59.743 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0051069510000000002, 2024-04-19T02:03:59.743 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.743 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.743 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.743 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.743 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.743 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.743 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.214412+0000", 2024-04-19T02:03:59.743 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.743 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.743 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.744 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.744 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.214412+0000", 2024-04-19T02:03:59.744 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.744 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.744 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.744 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.744 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.214413+0000", 2024-04-19T02:03:59.744 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.0999999999999997e-07 2024-04-19T02:03:59.744 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.744 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.744 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.744 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.215235+0000", 2024-04-19T02:03:59.745 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00082282700000000004 2024-04-19T02:03:59.745 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.745 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.745 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.745 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.215283+0000", 2024-04-19T02:03:59.745 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.7425000000000001e-05 2024-04-19T02:03:59.745 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.745 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.745 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.745 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.215308+0000", 2024-04-19T02:03:59.745 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5228999999999998e-05 2024-04-19T02:03:59.745 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.745 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.745 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.745 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.215526+0000", 2024-04-19T02:03:59.746 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000217512 2024-04-19T02:03:59.746 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.746 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.746 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.746 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.215559+0000", 2024-04-19T02:03:59.746 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3745000000000003e-05 2024-04-19T02:03:59.746 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.746 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.746 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.746 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.219423+0000", 2024-04-19T02:03:59.746 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0038636230000000001 2024-04-19T02:03:59.746 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.746 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.747 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.747 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.219519+0000", 2024-04-19T02:03:59.747 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5680000000000005e-05 2024-04-19T02:03:59.747 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.747 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.747 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.747 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.747 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.747 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.8s6 66/26 ECSubWrite(tid=150, reqid=client.4509.0:722, at_version=66'65, trim_to=0'0, roll_forward_to=57'63))", 2024-04-19T02:03:59.747 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:50.060942+0000", 2024-04-19T02:03:59.747 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 9.6110620440000005, 2024-04-19T02:03:59.747 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.005578838, 2024-04-19T02:03:59.747 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.748 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.748 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.748 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.748 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.748 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.748 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.060942+0000", 2024-04-19T02:03:59.748 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.748 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.748 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.748 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.749 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.060942+0000", 2024-04-19T02:03:59.749 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.749 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.749 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.749 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.749 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.060943+0000", 2024-04-19T02:03:59.749 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0299999999999998e-07 2024-04-19T02:03:59.749 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.749 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.749 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.749 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.061002+0000", 2024-04-19T02:03:59.749 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9414999999999999e-05 2024-04-19T02:03:59.749 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.749 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.749 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.750 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.061054+0000", 2024-04-19T02:03:59.750 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2048e-05 2024-04-19T02:03:59.750 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.750 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.750 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.750 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.061084+0000", 2024-04-19T02:03:59.750 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9102e-05 2024-04-19T02:03:59.750 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.750 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.750 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.750 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.061229+0000", 2024-04-19T02:03:59.750 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000144964 2024-04-19T02:03:59.750 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.750 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.750 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.751 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.061250+0000", 2024-04-19T02:03:59.751 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1874000000000001e-05 2024-04-19T02:03:59.751 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.751 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.751 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.751 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.066459+0000", 2024-04-19T02:03:59.751 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0052081799999999998 2024-04-19T02:03:59.751 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.751 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.751 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.751 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.066521+0000", 2024-04-19T02:03:59.751 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2551999999999993e-05 2024-04-19T02:03:59.751 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.751 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.751 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.752 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.752 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.752 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.8s6 66/26 ECSubWrite(tid=162, reqid=client.4509.0:752, at_version=66'71, trim_to=0'0, roll_forward_to=66'69))", 2024-04-19T02:03:59.752 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:50.394200+0000", 2024-04-19T02:03:59.752 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 9.2778042379999999, 2024-04-19T02:03:59.752 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0063256349999999996, 2024-04-19T02:03:59.752 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.752 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.752 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.752 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.752 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.752 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.752 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.394200+0000", 2024-04-19T02:03:59.752 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.752 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.753 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.753 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.753 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.394200+0000", 2024-04-19T02:03:59.753 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.753 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.753 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.753 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.753 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.394202+0000", 2024-04-19T02:03:59.753 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.883e-06 2024-04-19T02:03:59.753 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.753 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.753 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.753 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.394600+0000", 2024-04-19T02:03:59.753 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00039787600000000002 2024-04-19T02:03:59.753 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.753 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.754 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.754 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.394640+0000", 2024-04-19T02:03:59.754 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0561e-05 2024-04-19T02:03:59.754 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.754 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.754 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.754 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.394661+0000", 2024-04-19T02:03:59.754 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0925999999999999e-05 2024-04-19T02:03:59.754 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.754 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.754 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.754 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.394822+0000", 2024-04-19T02:03:59.754 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016112099999999999 2024-04-19T02:03:59.754 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.754 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.755 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.755 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.394848+0000", 2024-04-19T02:03:59.755 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5925999999999999e-05 2024-04-19T02:03:59.755 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.755 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.755 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.755 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.400443+0000", 2024-04-19T02:03:59.755 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0055950710000000001 2024-04-19T02:03:59.755 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.755 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.755 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.755 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.400526+0000", 2024-04-19T02:03:59.755 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.2270999999999999e-05 2024-04-19T02:03:59.755 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.755 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.755 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.756 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.756 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.756 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.5s6 67/26 ECSubWrite(tid=85, reqid=client.4509.0:765, at_version=67'41, trim_to=0'0, roll_forward_to=64'39))", 2024-04-19T02:03:59.756 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:51.097783+0000", 2024-04-19T02:03:59.756 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 8.5742214469999993, 2024-04-19T02:03:59.756 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0060386329999999998, 2024-04-19T02:03:59.756 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:03:59.756 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:03:59.756 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:03:59.756 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:03:59.756 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.756 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:03:59.756 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:51.097783+0000", 2024-04-19T02:03:59.756 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.756 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.757 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.757 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:03:59.757 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:51.097783+0000", 2024-04-19T02:03:59.757 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:03:59.757 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.757 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.757 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:03:59.757 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:51.097784+0000", 2024-04-19T02:03:59.757 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0559999999999999e-06 2024-04-19T02:03:59.757 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.757 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.757 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:03:59.757 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:51.098395+0000", 2024-04-19T02:03:59.757 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00061061600000000002 2024-04-19T02:03:59.757 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.757 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.758 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:03:59.758 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:51.098431+0000", 2024-04-19T02:03:59.758 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.6055e-05 2024-04-19T02:03:59.758 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.758 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.758 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:03:59.758 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:51.098455+0000", 2024-04-19T02:03:59.758 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4813000000000001e-05 2024-04-19T02:03:59.758 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.758 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.758 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:03:59.758 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:51.098650+0000", 2024-04-19T02:03:59.758 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00019473 2024-04-19T02:03:59.758 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.759 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.759 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:03:59.759 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:51.098680+0000", 2024-04-19T02:03:59.759 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9431999999999999e-05 2024-04-19T02:03:59.759 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.759 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.759 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:03:59.759 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:51.103758+0000", 2024-04-19T02:03:59.759 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.005078656 2024-04-19T02:03:59.759 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:03:59.759 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:03:59.759 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:03:59.759 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:51.103822+0000", 2024-04-19T02:03:59.759 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3275000000000007e-05 2024-04-19T02:03:59.759 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.759 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.760 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.760 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:03:59.760 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:03:59.760 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:59.761 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-19T02:03:59.762 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:03:59.761+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:03:59.863 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:03:59.861+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:03:59.890 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:03:59.890 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:03:59.890 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 0 2024-04-19T02:03:59.890 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:03:59.905 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-19T02:03:59.964 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:03:59.965+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:04:00.030 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:04:00.030 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:04:00.030 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-04-19T02:04:00.030 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-04-19T02:04:00.030 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:04:00.044 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-19T02:04:00.064 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:04:00.065+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:04:00.112 INFO:teuthology.orchestra.run.smithi059.stderr:nodeep-scrub is set 2024-04-19T02:04:00.117 INFO:tasks.rados.rados.0.smithi059.stdout:325: done (3 left) 2024-04-19T02:04:00.117 INFO:tasks.rados.rados.0.smithi059.stdout:326: done (2 left) 2024-04-19T02:04:00.117 INFO:tasks.rados.rados.0.smithi059.stdout:327: done (1 left) 2024-04-19T02:04:00.117 INFO:tasks.rados.rados.0.smithi059.stdout:328: done (0 left) 2024-04-19T02:04:00.118 INFO:tasks.rados.rados.0.smithi059.stdout:329: rollback oid 10 current snap is 28 2024-04-19T02:04:00.118 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 10 to 23 2024-04-19T02:04:00.118 INFO:tasks.rados.rados.0.smithi059.stdout:330: rollback oid 15 current snap is 28 2024-04-19T02:04:00.118 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 15 to 20 2024-04-19T02:04:00.118 INFO:tasks.rados.rados.0.smithi059.stdout:331: rollback oid 19 current snap is 28 2024-04-19T02:04:00.118 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 19 to 27 2024-04-19T02:04:00.118 INFO:tasks.rados.rados.0.smithi059.stdout:332: setattr oid 39 current snap is 28 2024-04-19T02:04:00.121 INFO:tasks.rados.rados.0.smithi059.stdout:333: copy_from oid 40 from oid 21 current snap is 28 2024-04-19T02:04:00.121 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 49 current snap is 28 2024-04-19T02:04:00.121 INFO:tasks.rados.rados.0.smithi059.stdout:334: seq_num 151 ranges {3817472=131072} 2024-04-19T02:04:00.147 INFO:tasks.rados.rados.0.smithi059.stdout:334: writing smithi05930959-49 from 3817472 to 3948544 tid 1 2024-04-19T02:04:00.147 INFO:tasks.rados.rados.0.smithi059.stdout:335: rollback oid 26 current snap is 28 2024-04-19T02:04:00.147 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 26 to 23 2024-04-19T02:04:00.147 INFO:tasks.rados.rados.0.smithi059.stdout:330: finishing rollback tid 1 to smithi05930959-15 2024-04-19T02:04:00.147 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 15 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-04-19T02:04:00.147 INFO:tasks.rados.rados.0.smithi059.stdout:330: done (6 left) 2024-04-19T02:04:00.147 INFO:tasks.rados.rados.0.smithi059.stdout:336: read oid 13 snap -1 2024-04-19T02:04:00.147 INFO:tasks.rados.rados.0.smithi059.stdout:336: expect (ObjNum 129 snap 19 seq_num 129) 2024-04-19T02:04:00.147 INFO:tasks.rados.rados.0.smithi059.stdout:337: read oid 35 snap -1 2024-04-19T02:04:00.147 INFO:tasks.rados.rados.0.smithi059.stdout:337: expect (ObjNum 34 snap 0 seq_num 34) 2024-04-19T02:04:00.147 INFO:tasks.rados.rados.0.smithi059.stdout:338: copy_from oid 41 from oid 18 current snap is 28 2024-04-19T02:04:00.147 INFO:tasks.rados.rados.0.smithi059.stdout:331: finishing rollback tid 1 to smithi05930959-19 2024-04-19T02:04:00.147 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 19 v 63 (ObjNum 81 snap 0 seq_num 533364283) dirty dne 2024-04-19T02:04:00.148 INFO:tasks.rados.rados.0.smithi059.stdout:329: finishing rollback tid 0 to smithi05930959-10 2024-04-19T02:04:00.148 INFO:tasks.rados.rados.0.smithi059.stdout:331: done (8 left) 2024-04-19T02:04:00.148 INFO:tasks.rados.rados.0.smithi059.stdout:339: setattr oid 43 current snap is 28 2024-04-19T02:04:00.148 INFO:tasks.rados.rados.0.smithi059.stdout:329: finishing rollback tid 1 to smithi05930959-10 2024-04-19T02:04:00.148 INFO:tasks.rados.rados.0.smithi059.stdout:329: finishing rollback tid 2 to smithi05930959-10 2024-04-19T02:04:00.148 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 10 v 151 (ObjNum 88 snap 9 seq_num 88) dirty exists 2024-04-19T02:04:00.148 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 39 v 97 (ObjNum 144 snap 24 seq_num 144) dirty exists 2024-04-19T02:04:00.149 INFO:tasks.rados.rados.0.smithi059.stdout:329: done (8 left) 2024-04-19T02:04:00.150 INFO:tasks.rados.rados.0.smithi059.stdout:332: done (7 left) 2024-04-19T02:04:00.150 INFO:tasks.rados.rados.0.smithi059.stdout:340: rmattr oid 42 current snap is 28 2024-04-19T02:04:00.150 INFO:tasks.rados.rados.0.smithi059.stdout:340: done (7 left) 2024-04-19T02:04:00.150 INFO:tasks.rados.rados.0.smithi059.stdout:341: rollback oid 32 current snap is 28 2024-04-19T02:04:00.150 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 32 to 10 2024-04-19T02:04:00.150 INFO:tasks.rados.rados.0.smithi059.stdout:342: setattr oid 36 current snap is 28 2024-04-19T02:04:00.153 INFO:tasks.rados.rados.0.smithi059.stdout:343: read oid 42 snap 20 2024-04-19T02:04:00.153 INFO:tasks.rados.rados.0.smithi059.stdout:343: expect (ObjNum 117 snap 17 seq_num 117) 2024-04-19T02:04:00.153 INFO:tasks.rados.rados.0.smithi059.stdout:344: setattr oid 10 current snap is 28 2024-04-19T02:04:00.156 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 4 current snap is 28 2024-04-19T02:04:00.156 INFO:tasks.rados.rados.0.smithi059.stdout:345: seq_num 152 ranges {3899392=147456} 2024-04-19T02:04:00.165 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:04:00.165+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:04:00.180 INFO:tasks.rados.rados.0.smithi059.stdout:345: writing smithi05930959-4 from 3899392 to 4046848 tid 1 2024-04-19T02:04:00.181 INFO:tasks.rados.rados.0.smithi059.stdout:346: read oid 7 snap -1 2024-04-19T02:04:00.181 INFO:tasks.rados.rados.0.smithi059.stdout:346: expect (ObjNum 91 snap 9 seq_num 91) 2024-04-19T02:04:00.181 INFO:tasks.rados.rados.0.smithi059.stdout:347: snap_create 2024-04-19T02:04:00.181 INFO:tasks.rados.rados.0.smithi059.stdout:335: finishing rollback tid 0 to smithi05930959-26 2024-04-19T02:04:00.181 INFO:tasks.rados.rados.0.smithi059.stdout:333: finishing copy_from racing read to smithi05930959-40 2024-04-19T02:04:00.181 INFO:tasks.rados.rados.0.smithi059.stdout:337: expect (ObjNum 34 snap 0 seq_num 34) 2024-04-19T02:04:00.183 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:04:00.183 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-04-19T02:04:00.183 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-04-19T02:04:00.183 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-04-19T02:04:00.183 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.183 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:355 3.es0 3:7af77758:::smithi05930959-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:04:00.183 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:24.772697+0000", 2024-04-19T02:04:00.184 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 35.410988371999998, 2024-04-19T02:04:00.184 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014504697, 2024-04-19T02:04:00.184 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:00.184 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:00.184 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:04:00.184 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:04:00.184 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:04:00.184 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:00.184 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 355 2024-04-19T02:04:00.184 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.184 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:00.184 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.184 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:00.184 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.772697+0000", 2024-04-19T02:04:00.185 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.185 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.185 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.185 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:00.185 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.772697+0000", 2024-04-19T02:04:00.185 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.185 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.185 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.185 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:00.185 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.772699+0000", 2024-04-19T02:04:00.185 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3430000000000001e-06 2024-04-19T02:04:00.186 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.186 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.186 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:00.186 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.772706+0000", 2024-04-19T02:04:00.186 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2400000000000001e-06 2024-04-19T02:04:00.186 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.186 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.186 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:00.186 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.772724+0000", 2024-04-19T02:04:00.186 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8289000000000001e-05 2024-04-19T02:04:00.186 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.186 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.186 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.186 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.772738+0000", 2024-04-19T02:04:00.187 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3732e-05 2024-04-19T02:04:00.187 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.187 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.187 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:00.187 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.784149+0000", 2024-04-19T02:04:00.187 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011411361 2024-04-19T02:04:00.187 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.188 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.188 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:04:00.188 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.784288+0000", 2024-04-19T02:04:00.188 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013880699999999999 2024-04-19T02:04:00.188 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.188 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.188 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:00.188 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:24.787202+0000", 2024-04-19T02:04:00.188 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0029139249999999999 2024-04-19T02:04:00.188 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.188 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:00.189 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.189 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.189 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.189 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.bs7 42/26 ECSubWrite(tid=133, reqid=osd.3.0:133, at_version=42'21, trim_to=0'0, roll_forward_to=39'19))", 2024-04-19T02:04:00.189 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.748986+0000", 2024-04-19T02:04:00.189 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 33.434699201999997, 2024-04-19T02:04:00.189 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.015850517000000001, 2024-04-19T02:04:00.189 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:00.189 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:00.189 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:00.189 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:00.189 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.190 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:00.190 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748986+0000", 2024-04-19T02:04:00.190 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.190 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.190 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.190 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:00.190 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748986+0000", 2024-04-19T02:04:00.190 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.190 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.190 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.190 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:00.191 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.748987+0000", 2024-04-19T02:04:00.191 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.6000000000000003e-07 2024-04-19T02:04:00.191 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.191 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.191 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:00.191 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749579+0000", 2024-04-19T02:04:00.191 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00059223999999999995 2024-04-19T02:04:00.191 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.191 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.191 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:00.191 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749611+0000", 2024-04-19T02:04:00.191 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1554999999999999e-05 2024-04-19T02:04:00.191 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.191 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.192 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.192 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749633+0000", 2024-04-19T02:04:00.192 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1885000000000001e-05 2024-04-19T02:04:00.192 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.192 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.192 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:00.192 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749799+0000", 2024-04-19T02:04:00.192 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000166422 2024-04-19T02:04:00.192 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.192 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.192 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:00.192 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749825+0000", 2024-04-19T02:04:00.192 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5825000000000001e-05 2024-04-19T02:04:00.192 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.192 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.193 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:00.193 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.764686+0000", 2024-04-19T02:04:00.193 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014861087 2024-04-19T02:04:00.193 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.193 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.193 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:00.193 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.764837+0000", 2024-04-19T02:04:00.193 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000150743 2024-04-19T02:04:00.193 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.193 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:00.193 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.193 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.193 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.193 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.bs7 42/26 ECSubWrite(tid=135, reqid=osd.3.0:135, at_version=42'23, trim_to=0'0, roll_forward_to=39'19))", 2024-04-19T02:04:00.194 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.749928+0000", 2024-04-19T02:04:00.194 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 33.433757817999997, 2024-04-19T02:04:00.194 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.017644765, 2024-04-19T02:04:00.194 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:00.194 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:00.194 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:00.194 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:00.194 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.194 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:00.194 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749928+0000", 2024-04-19T02:04:00.194 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.194 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.194 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.194 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:00.195 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749928+0000", 2024-04-19T02:04:00.195 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.195 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.195 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.195 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:00.195 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749929+0000", 2024-04-19T02:04:00.195 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.136e-06 2024-04-19T02:04:00.195 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.195 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.195 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:00.195 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749999+0000", 2024-04-19T02:04:00.195 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0471999999999999e-05 2024-04-19T02:04:00.195 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.195 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.195 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:00.196 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.750024+0000", 2024-04-19T02:04:00.196 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4575000000000001e-05 2024-04-19T02:04:00.196 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.196 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.196 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.196 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.750044+0000", 2024-04-19T02:04:00.196 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9802000000000001e-05 2024-04-19T02:04:00.196 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.196 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.196 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:00.196 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.756068+0000", 2024-04-19T02:04:00.196 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0060238050000000001 2024-04-19T02:04:00.196 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.196 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.197 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:00.197 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.756097+0000", 2024-04-19T02:04:00.197 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9045000000000001e-05 2024-04-19T02:04:00.197 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.197 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.197 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:00.197 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.767487+0000", 2024-04-19T02:04:00.197 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011390612 2024-04-19T02:04:00.197 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.197 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.197 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:00.197 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.767573+0000", 2024-04-19T02:04:00.197 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5317999999999999e-05 2024-04-19T02:04:00.197 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.198 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:00.198 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.198 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.198 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.198 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.ds2 46/26 ECSubWrite(tid=187, reqid=osd.8.0:187, at_version=46'35, trim_to=0'0, roll_forward_to=45'31))", 2024-04-19T02:04:00.198 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.058580+0000", 2024-04-19T02:04:00.198 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 30.125105720000001, 2024-04-19T02:04:00.198 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.016482007999999999, 2024-04-19T02:04:00.198 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:00.198 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:00.198 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:00.198 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:00.198 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.198 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:00.199 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.058580+0000", 2024-04-19T02:04:00.199 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.199 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.199 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.199 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:00.199 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.058580+0000", 2024-04-19T02:04:00.199 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.199 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.199 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.199 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:00.199 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.058582+0000", 2024-04-19T02:04:00.199 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7069999999999999e-06 2024-04-19T02:04:00.199 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.199 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.200 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:00.200 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.058632+0000", 2024-04-19T02:04:00.200 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0710000000000001e-05 2024-04-19T02:04:00.200 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.200 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.200 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:00.200 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.058664+0000", 2024-04-19T02:04:00.200 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1448000000000001e-05 2024-04-19T02:04:00.200 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.200 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.200 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.200 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.058688+0000", 2024-04-19T02:04:00.200 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4405000000000001e-05 2024-04-19T02:04:00.200 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.201 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.201 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:00.201 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.059171+0000", 2024-04-19T02:04:00.201 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00048236699999999998 2024-04-19T02:04:00.201 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.201 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.201 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:00.201 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.059195+0000", 2024-04-19T02:04:00.201 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4042000000000001e-05 2024-04-19T02:04:00.201 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.201 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.201 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:00.201 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.074940+0000", 2024-04-19T02:04:00.202 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.015745333 2024-04-19T02:04:00.202 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.202 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.202 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:00.202 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.075062+0000", 2024-04-19T02:04:00.202 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000121996 2024-04-19T02:04:00.202 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.202 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:00.202 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.202 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.202 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.202 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.4s4 46/26 ECSubWrite(tid=58, reqid=osd.1.0:58, at_version=46'44, trim_to=0'0, roll_forward_to=45'42))", 2024-04-19T02:04:00.202 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.059794+0000", 2024-04-19T02:04:00.203 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 30.123892041000001, 2024-04-19T02:04:00.203 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.015185597, 2024-04-19T02:04:00.203 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:00.203 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:00.203 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:00.203 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:00.203 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.203 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:00.203 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.059794+0000", 2024-04-19T02:04:00.203 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.203 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.203 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.203 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:00.203 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.059794+0000", 2024-04-19T02:04:00.204 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.204 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.204 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.204 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:00.204 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.059794+0000", 2024-04-19T02:04:00.204 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2399999999999998e-07 2024-04-19T02:04:00.204 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.204 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.204 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:00.204 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.059843+0000", 2024-04-19T02:04:00.204 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9217e-05 2024-04-19T02:04:00.204 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.204 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.204 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:00.205 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.059888+0000", 2024-04-19T02:04:00.205 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4842999999999997e-05 2024-04-19T02:04:00.205 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.205 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.205 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.205 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.059911+0000", 2024-04-19T02:04:00.205 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2829000000000001e-05 2024-04-19T02:04:00.205 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.205 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.205 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:00.205 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.060084+0000", 2024-04-19T02:04:00.205 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017278000000000001 2024-04-19T02:04:00.205 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.205 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.206 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:00.206 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.060110+0000", 2024-04-19T02:04:00.206 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5931999999999998e-05 2024-04-19T02:04:00.206 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.206 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.206 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:00.206 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.074866+0000", 2024-04-19T02:04:00.206 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014756136 2024-04-19T02:04:00.206 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.206 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.206 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:00.206 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.074979+0000", 2024-04-19T02:04:00.206 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000113236 2024-04-19T02:04:00.206 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.206 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:00.207 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.207 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.207 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.207 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.as5 46/26 ECSubWrite(tid=102, reqid=osd.11.0:102, at_version=46'47, trim_to=0'0, roll_forward_to=45'45))", 2024-04-19T02:04:00.207 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.066484+0000", 2024-04-19T02:04:00.207 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 30.117202057, 2024-04-19T02:04:00.207 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014065139000000001, 2024-04-19T02:04:00.207 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:00.207 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:00.207 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:00.207 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:00.207 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.207 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:00.207 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066484+0000", 2024-04-19T02:04:00.208 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.208 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.208 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.208 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:00.208 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066484+0000", 2024-04-19T02:04:00.208 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.208 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.208 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.208 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:00.208 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066484+0000", 2024-04-19T02:04:00.208 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0299999999999998e-07 2024-04-19T02:04:00.208 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.208 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.208 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:00.208 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066542+0000", 2024-04-19T02:04:00.209 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.7182999999999998e-05 2024-04-19T02:04:00.209 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.209 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.209 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:00.209 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066572+0000", 2024-04-19T02:04:00.209 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0205e-05 2024-04-19T02:04:00.209 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.209 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.209 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.209 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066594+0000", 2024-04-19T02:04:00.209 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2478999999999999e-05 2024-04-19T02:04:00.209 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.209 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.209 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:00.210 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066759+0000", 2024-04-19T02:04:00.210 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016516999999999999 2024-04-19T02:04:00.210 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.210 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.210 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:00.210 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066784+0000", 2024-04-19T02:04:00.210 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4587999999999999e-05 2024-04-19T02:04:00.210 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.210 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.210 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:00.210 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.080462+0000", 2024-04-19T02:04:00.210 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013678459 2024-04-19T02:04:00.210 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.210 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.210 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:00.211 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.080549+0000", 2024-04-19T02:04:00.211 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.6252e-05 2024-04-19T02:04:00.211 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.211 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:00.211 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.211 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.211 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.211 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.cs3 46/26 ECSubWrite(tid=44, reqid=osd.9.0:44, at_version=46'26, trim_to=0'0, roll_forward_to=45'24))", 2024-04-19T02:04:00.211 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.066755+0000", 2024-04-19T02:04:00.211 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 30.116930289999999, 2024-04-19T02:04:00.211 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.01420738, 2024-04-19T02:04:00.211 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:00.211 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:00.211 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:00.211 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:00.212 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.212 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:00.212 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066755+0000", 2024-04-19T02:04:00.212 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.212 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.212 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.212 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:00.212 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066755+0000", 2024-04-19T02:04:00.212 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.212 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.212 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.212 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:00.212 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066756+0000", 2024-04-19T02:04:00.212 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4000000000000002e-07 2024-04-19T02:04:00.213 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.213 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.213 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:00.213 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066799+0000", 2024-04-19T02:04:00.213 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3217000000000003e-05 2024-04-19T02:04:00.213 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.213 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.213 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:00.213 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066825+0000", 2024-04-19T02:04:00.213 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6248999999999999e-05 2024-04-19T02:04:00.213 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.213 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.213 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.213 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066851+0000", 2024-04-19T02:04:00.214 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5171e-05 2024-04-19T02:04:00.214 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.214 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.214 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:00.214 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.067016+0000", 2024-04-19T02:04:00.214 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016518699999999999 2024-04-19T02:04:00.214 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.214 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.214 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:00.214 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.067042+0000", 2024-04-19T02:04:00.214 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5992000000000001e-05 2024-04-19T02:04:00.214 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.214 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.214 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:00.215 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.080884+0000", 2024-04-19T02:04:00.215 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013842047999999999 2024-04-19T02:04:00.215 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.215 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.215 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:00.215 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.080963+0000", 2024-04-19T02:04:00.215 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.8975999999999997e-05 2024-04-19T02:04:00.215 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.215 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:00.215 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.215 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.215 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.215 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.as5 46/26 ECSubWrite(tid=103, reqid=osd.11.0:103, at_version=46'49, trim_to=0'0, roll_forward_to=45'45))", 2024-04-19T02:04:00.215 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.067427+0000", 2024-04-19T02:04:00.215 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 30.116258125000002, 2024-04-19T02:04:00.215 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.020157152000000001, 2024-04-19T02:04:00.216 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:00.216 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:00.216 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:00.216 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:00.216 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.216 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:00.216 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.067427+0000", 2024-04-19T02:04:00.216 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.216 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.216 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.216 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:00.216 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.067427+0000", 2024-04-19T02:04:00.216 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.216 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.217 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.217 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:00.217 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.067429+0000", 2024-04-19T02:04:00.217 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7439999999999999e-06 2024-04-19T02:04:00.217 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.217 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.217 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:00.217 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.067489+0000", 2024-04-19T02:04:00.217 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9988999999999998e-05 2024-04-19T02:04:00.217 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.217 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.217 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:00.217 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.068116+0000", 2024-04-19T02:04:00.217 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00062675199999999995 2024-04-19T02:04:00.217 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.218 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.218 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.218 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.068146+0000", 2024-04-19T02:04:00.218 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9751999999999999e-05 2024-04-19T02:04:00.218 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.218 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.218 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:00.218 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.070935+0000", 2024-04-19T02:04:00.218 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0027894399999999998 2024-04-19T02:04:00.218 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.218 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.218 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:00.218 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.070950+0000", 2024-04-19T02:04:00.219 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5218999999999999e-05 2024-04-19T02:04:00.219 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.219 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.219 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:00.219 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.087492+0000", 2024-04-19T02:04:00.219 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.016541891999999999 2024-04-19T02:04:00.219 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.219 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.219 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:00.219 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.087585+0000", 2024-04-19T02:04:00.219 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.2363999999999999e-05 2024-04-19T02:04:00.219 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.219 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:00.219 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.219 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.220 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.220 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.7s4 46/26 ECSubWrite(tid=158, reqid=osd.3.0:158, at_version=46'14, trim_to=0'0, roll_forward_to=45'12))", 2024-04-19T02:04:00.220 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.072853+0000", 2024-04-19T02:04:00.220 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 30.110832127999998, 2024-04-19T02:04:00.220 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014228316, 2024-04-19T02:04:00.220 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:00.220 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:00.220 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:00.220 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:00.220 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.220 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:00.220 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072853+0000", 2024-04-19T02:04:00.220 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.220 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.221 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.221 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:00.221 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072853+0000", 2024-04-19T02:04:00.221 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.221 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.221 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.221 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:00.221 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072854+0000", 2024-04-19T02:04:00.221 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5700000000000002e-07 2024-04-19T02:04:00.221 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.221 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.221 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:00.221 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072905+0000", 2024-04-19T02:04:00.221 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1443000000000002e-05 2024-04-19T02:04:00.222 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.222 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.222 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:00.222 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072931+0000", 2024-04-19T02:04:00.222 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5103e-05 2024-04-19T02:04:00.222 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.222 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.222 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.222 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.072953+0000", 2024-04-19T02:04:00.222 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2174000000000001e-05 2024-04-19T02:04:00.222 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.222 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.222 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:00.222 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.073226+0000", 2024-04-19T02:04:00.223 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000273174 2024-04-19T02:04:00.223 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.223 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.223 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:00.223 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.073252+0000", 2024-04-19T02:04:00.223 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6152e-05 2024-04-19T02:04:00.223 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.223 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.223 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:00.223 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.086987+0000", 2024-04-19T02:04:00.223 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013734837999999999 2024-04-19T02:04:00.223 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.223 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.223 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:00.224 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.087082+0000", 2024-04-19T02:04:00.224 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4875000000000002e-05 2024-04-19T02:04:00.224 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.224 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:00.224 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.224 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.224 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.224 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.4s4 46/26 ECSubWrite(tid=59, reqid=osd.1.0:58, at_version=0'0, trim_to=0'0, roll_forward_to=46'44))", 2024-04-19T02:04:00.224 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.076912+0000", 2024-04-19T02:04:00.224 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 30.106773207, 2024-04-19T02:04:00.224 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014831941, 2024-04-19T02:04:00.224 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:00.224 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:00.224 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:00.224 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:00.225 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.225 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:00.225 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.076912+0000", 2024-04-19T02:04:00.225 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.225 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.225 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.225 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:00.225 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.076912+0000", 2024-04-19T02:04:00.225 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.225 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.225 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.225 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:00.225 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.076914+0000", 2024-04-19T02:04:00.225 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.829e-06 2024-04-19T02:04:00.226 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.226 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.226 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:00.226 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.076944+0000", 2024-04-19T02:04:00.226 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9482e-05 2024-04-19T02:04:00.226 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.226 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.226 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:00.226 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.080072+0000", 2024-04-19T02:04:00.226 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0031280629999999999 2024-04-19T02:04:00.226 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.226 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.226 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.227 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.080101+0000", 2024-04-19T02:04:00.227 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9524999999999999e-05 2024-04-19T02:04:00.227 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.227 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.227 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:00.227 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.081114+0000", 2024-04-19T02:04:00.227 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0010130250000000001 2024-04-19T02:04:00.227 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.227 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.227 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:00.227 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.081138+0000", 2024-04-19T02:04:00.227 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4105e-05 2024-04-19T02:04:00.227 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.227 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.228 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:00.228 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.091652+0000", 2024-04-19T02:04:00.228 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010514055 2024-04-19T02:04:00.228 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.228 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.228 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:00.228 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.091744+0000", 2024-04-19T02:04:00.228 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1856999999999998e-05 2024-04-19T02:04:00.228 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.228 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:00.228 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.228 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.228 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.228 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:420 3.es0 3:7831c41d:::smithi05930959-30:head [copy-from ver 14] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:00.229 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.193498+0000", 2024-04-19T02:04:00.229 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 29.990187531, 2024-04-19T02:04:00.229 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.052261775000000003, 2024-04-19T02:04:00.229 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:00.229 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:00.229 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:00.229 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:04:00.229 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:04:00.229 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:00.229 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 420 2024-04-19T02:04:00.229 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.229 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:00.229 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.229 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:00.229 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193498+0000", 2024-04-19T02:04:00.230 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.230 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.230 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.230 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:00.230 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193498+0000", 2024-04-19T02:04:00.230 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.230 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.230 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.230 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:00.230 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193501+0000", 2024-04-19T02:04:00.230 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.88e-06 2024-04-19T02:04:00.230 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.230 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.231 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:00.231 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193514+0000", 2024-04-19T02:04:00.231 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2879e-05 2024-04-19T02:04:00.231 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.231 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.231 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:00.231 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193546+0000", 2024-04-19T02:04:00.231 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1931999999999999e-05 2024-04-19T02:04:00.231 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.231 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.231 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.231 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193560+0000", 2024-04-19T02:04:00.231 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4609e-05 2024-04-19T02:04:00.231 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.231 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.232 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:00.232 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193695+0000", 2024-04-19T02:04:00.232 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013434499999999999 2024-04-19T02:04:00.232 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.232 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.232 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:04:00.232 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193864+0000", 2024-04-19T02:04:00.232 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016894300000000001 2024-04-19T02:04:00.232 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.232 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.232 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:00.232 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.239884+0000", 2024-04-19T02:04:00.232 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.046020559000000003 2024-04-19T02:04:00.232 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.233 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.233 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:00.233 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.245339+0000", 2024-04-19T02:04:00.233 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0054546569999999999 2024-04-19T02:04:00.233 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.233 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.233 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:04:00.233 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.245619+0000", 2024-04-19T02:04:00.233 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00028016800000000002 2024-04-19T02:04:00.233 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.233 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.233 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:00.233 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.245760+0000", 2024-04-19T02:04:00.233 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000140803 2024-04-19T02:04:00.234 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.234 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:00.234 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.234 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.234 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.234 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:421 3.es0 3:7831c41d:::smithi05930959-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:00.234 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.193593+0000", 2024-04-19T02:04:00.234 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 29.990093027, 2024-04-19T02:04:00.234 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.049216688000000001, 2024-04-19T02:04:00.234 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:00.234 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:00.234 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:04:00.234 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:04:00.234 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:04:00.235 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:00.235 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 421 2024-04-19T02:04:00.235 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.235 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:00.235 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.235 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:00.235 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193593+0000", 2024-04-19T02:04:00.235 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.235 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.235 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.235 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:00.235 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193593+0000", 2024-04-19T02:04:00.235 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.235 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.236 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.236 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:00.236 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193594+0000", 2024-04-19T02:04:00.236 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4169999999999999e-06 2024-04-19T02:04:00.236 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.236 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.236 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:00.236 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193601+0000", 2024-04-19T02:04:00.236 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6429999999999998e-06 2024-04-19T02:04:00.236 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.236 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.236 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:00.236 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193621+0000", 2024-04-19T02:04:00.236 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0435e-05 2024-04-19T02:04:00.236 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.237 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.237 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.237 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193636+0000", 2024-04-19T02:04:00.237 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4822e-05 2024-04-19T02:04:00.237 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.237 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.237 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:00.237 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.194259+0000", 2024-04-19T02:04:00.237 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00062306299999999998 2024-04-19T02:04:00.237 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.237 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.237 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for blocked object", 2024-04-19T02:04:00.237 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.194433+0000", 2024-04-19T02:04:00.237 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017404899999999999 2024-04-19T02:04:00.238 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.238 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.238 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:00.238 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.242239+0000", 2024-04-19T02:04:00.238 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.047806081 2024-04-19T02:04:00.238 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.238 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.238 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:04:00.238 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.242500+0000", 2024-04-19T02:04:00.238 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00026063699999999998 2024-04-19T02:04:00.238 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.238 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.238 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:00.239 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.242809+0000", 2024-04-19T02:04:00.239 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00030954099999999998 2024-04-19T02:04:00.239 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.239 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:00.239 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.239 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.239 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.239 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.bs7 47/26 ECSubWrite(tid=183, reqid=osd.3.0:183, at_version=47'38, trim_to=0'0, roll_forward_to=46'34))", 2024-04-19T02:04:00.239 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.084378+0000", 2024-04-19T02:04:00.239 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 29.099307939999999, 2024-04-19T02:04:00.239 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.020336838999999999, 2024-04-19T02:04:00.239 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:00.239 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:00.239 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:00.239 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:00.240 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.240 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:00.240 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.084378+0000", 2024-04-19T02:04:00.240 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.240 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.240 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.240 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:00.240 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.084378+0000", 2024-04-19T02:04:00.240 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.240 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.240 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.240 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:00.240 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.084379+0000", 2024-04-19T02:04:00.241 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3230000000000001e-06 2024-04-19T02:04:00.241 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.241 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.241 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:00.241 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.084443+0000", 2024-04-19T02:04:00.241 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3745999999999996e-05 2024-04-19T02:04:00.241 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.241 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.241 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:00.241 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.084476+0000", 2024-04-19T02:04:00.241 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3352000000000002e-05 2024-04-19T02:04:00.241 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.241 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.241 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.242 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.084500+0000", 2024-04-19T02:04:00.242 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3492000000000001e-05 2024-04-19T02:04:00.242 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.242 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.242 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:00.242 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.084742+0000", 2024-04-19T02:04:00.242 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00024195900000000001 2024-04-19T02:04:00.242 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.242 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.242 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:00.242 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.084769+0000", 2024-04-19T02:04:00.242 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7222000000000001e-05 2024-04-19T02:04:00.242 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.243 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.243 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:00.243 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.104626+0000", 2024-04-19T02:04:00.243 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.019857593999999999 2024-04-19T02:04:00.243 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.243 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.243 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:00.243 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.104715+0000", 2024-04-19T02:04:00.243 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8151000000000004e-05 2024-04-19T02:04:00.243 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.243 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:00.243 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.244 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.244 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.244 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.1s5 47/26 ECSubWrite(tid=30, reqid=osd.0.0:30, at_version=47'17, trim_to=0'0, roll_forward_to=46'13))", 2024-04-19T02:04:00.244 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.085073+0000", 2024-04-19T02:04:00.244 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 29.098612915, 2024-04-19T02:04:00.244 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.019688054999999999, 2024-04-19T02:04:00.244 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:00.244 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:00.244 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:00.244 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:00.244 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.244 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:00.244 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085073+0000", 2024-04-19T02:04:00.244 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.245 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.245 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.245 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:00.245 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085073+0000", 2024-04-19T02:04:00.245 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.245 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.245 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.245 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:00.245 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085075+0000", 2024-04-19T02:04:00.245 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.81e-06 2024-04-19T02:04:00.245 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.245 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.245 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:00.245 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085126+0000", 2024-04-19T02:04:00.245 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1100000000000002e-05 2024-04-19T02:04:00.246 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.246 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.246 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:00.246 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085158+0000", 2024-04-19T02:04:00.246 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.2375000000000001e-05 2024-04-19T02:04:00.246 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.246 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.246 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.246 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085180+0000", 2024-04-19T02:04:00.246 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1942000000000001e-05 2024-04-19T02:04:00.246 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.246 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.246 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:00.247 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.086348+0000", 2024-04-19T02:04:00.247 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001167757 2024-04-19T02:04:00.247 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.247 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.247 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:00.247 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.086362+0000", 2024-04-19T02:04:00.247 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4749000000000001e-05 2024-04-19T02:04:00.247 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.247 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.247 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:00.247 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.104692+0000", 2024-04-19T02:04:00.247 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.018329525999999999 2024-04-19T02:04:00.247 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.247 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.247 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:00.247 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.104761+0000", 2024-04-19T02:04:00.248 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.8795999999999997e-05 2024-04-19T02:04:00.248 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.248 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:00.248 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.248 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.248 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.248 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.8s2 47/26 ECSubWrite(tid=86, reqid=osd.4.0:86, at_version=47'44, trim_to=0'0, roll_forward_to=46'42))", 2024-04-19T02:04:00.248 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.093211+0000", 2024-04-19T02:04:00.248 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 29.090474857, 2024-04-19T02:04:00.248 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014295323, 2024-04-19T02:04:00.248 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:00.248 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:00.248 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:00.248 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:00.248 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.249 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:00.249 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.093211+0000", 2024-04-19T02:04:00.249 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.249 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.249 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.249 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:00.249 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.093211+0000", 2024-04-19T02:04:00.249 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.249 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.249 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.249 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:00.249 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.093211+0000", 2024-04-19T02:04:00.249 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.3900000000000004e-07 2024-04-19T02:04:00.249 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.250 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.250 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:00.250 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.093266+0000", 2024-04-19T02:04:00.250 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4684e-05 2024-04-19T02:04:00.250 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.250 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.250 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:00.250 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.093292+0000", 2024-04-19T02:04:00.250 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6001999999999999e-05 2024-04-19T02:04:00.250 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.250 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.250 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.250 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.093308+0000", 2024-04-19T02:04:00.250 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6052000000000001e-05 2024-04-19T02:04:00.251 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.251 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.251 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:00.251 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.093541+0000", 2024-04-19T02:04:00.251 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00023266000000000001 2024-04-19T02:04:00.251 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.251 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.251 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:00.251 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.093573+0000", 2024-04-19T02:04:00.251 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1908e-05 2024-04-19T02:04:00.251 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.251 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.251 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:00.251 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.107421+0000", 2024-04-19T02:04:00.251 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013848035999999999 2024-04-19T02:04:00.252 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.252 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.252 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:00.252 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.107506+0000", 2024-04-19T02:04:00.252 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5341999999999997e-05 2024-04-19T02:04:00.252 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.252 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:00.252 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.252 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.252 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.252 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:698 3.es0 3:79fe7502:::smithi05930959-12:head [append 0~753664] snapc 14=[14,13,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-19T02:04:00.252 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:46.461930+0000", 2024-04-19T02:04:00.252 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 13.721755627, 2024-04-19T02:04:00.253 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.016145176000000001, 2024-04-19T02:04:00.253 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:00.253 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:00.253 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:00.253 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:04:00.253 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:04:00.253 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:00.253 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 698 2024-04-19T02:04:00.253 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.253 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:00.253 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.253 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:00.253 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.461930+0000", 2024-04-19T02:04:00.253 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.253 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.253 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.254 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:00.254 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.461930+0000", 2024-04-19T02:04:00.254 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.254 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.254 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.254 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:00.254 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.461953+0000", 2024-04-19T02:04:00.254 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3332e-05 2024-04-19T02:04:00.254 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.254 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.254 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:00.254 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.463352+0000", 2024-04-19T02:04:00.254 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0013982770000000001 2024-04-19T02:04:00.254 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.255 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.255 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:00.255 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.463391+0000", 2024-04-19T02:04:00.255 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9038999999999997e-05 2024-04-19T02:04:00.255 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.255 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.255 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.255 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.463415+0000", 2024-04-19T02:04:00.255 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4362000000000002e-05 2024-04-19T02:04:00.255 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.255 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.255 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:00.255 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.463690+0000", 2024-04-19T02:04:00.255 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00027526599999999998 2024-04-19T02:04:00.255 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.255 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.255 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:04:00.256 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.463960+0000", 2024-04-19T02:04:00.256 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00027012899999999997 2024-04-19T02:04:00.256 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.256 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.256 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:00.256 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.466757+0000", 2024-04-19T02:04:00.256 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0027967280000000001 2024-04-19T02:04:00.256 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.256 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.256 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:00.256 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.477367+0000", 2024-04-19T02:04:00.256 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010610029 2024-04-19T02:04:00.256 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.256 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.256 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:04:00.256 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.477923+0000", 2024-04-19T02:04:00.257 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00055550200000000004 2024-04-19T02:04:00.257 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.257 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.257 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:00.257 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.478075+0000", 2024-04-19T02:04:00.257 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015251199999999999 2024-04-19T02:04:00.257 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.257 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:00.257 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.257 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.257 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.257 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:699 3.es0 3:79fe7502:::smithi05930959-12:head [append 0~475136] snapc 14=[14,13,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-19T02:04:00.257 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:46.464793+0000", 2024-04-19T02:04:00.257 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 13.718892478000001, 2024-04-19T02:04:00.257 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014170390999999999, 2024-04-19T02:04:00.257 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:00.257 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:00.258 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:00.258 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:04:00.258 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:04:00.258 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:00.258 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 699 2024-04-19T02:04:00.258 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.258 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:00.258 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.258 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:00.258 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.464793+0000", 2024-04-19T02:04:00.258 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.258 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.258 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.258 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:00.259 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.464793+0000", 2024-04-19T02:04:00.259 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.259 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.259 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.259 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:00.259 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.464794+0000", 2024-04-19T02:04:00.259 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3570000000000001e-06 2024-04-19T02:04:00.259 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.259 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.259 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:00.259 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.465189+0000", 2024-04-19T02:04:00.259 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00039490599999999999 2024-04-19T02:04:00.259 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.259 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.259 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:00.259 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.465204+0000", 2024-04-19T02:04:00.260 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4174e-05 2024-04-19T02:04:00.260 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.260 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.260 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.260 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.465212+0000", 2024-04-19T02:04:00.260 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3650000000000004e-06 2024-04-19T02:04:00.260 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.260 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.260 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:00.260 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.468821+0000", 2024-04-19T02:04:00.260 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0036091640000000002 2024-04-19T02:04:00.260 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.260 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.260 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:04:00.260 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.468928+0000", 2024-04-19T02:04:00.261 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000106697 2024-04-19T02:04:00.261 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.261 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.261 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:00.261 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.469948+0000", 2024-04-19T02:04:00.261 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0010202150000000001 2024-04-19T02:04:00.261 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.261 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.261 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:00.261 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.477369+0000", 2024-04-19T02:04:00.261 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.007420729 2024-04-19T02:04:00.261 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.261 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.261 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:04:00.261 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.478802+0000", 2024-04-19T02:04:00.261 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0014337390000000001 2024-04-19T02:04:00.261 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.262 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.262 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:00.262 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.478964+0000", 2024-04-19T02:04:00.262 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016104499999999999 2024-04-19T02:04:00.262 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.262 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:00.262 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.262 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.262 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.262 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:700 3.es0 3:79fe7502:::smithi05930959-12:head [append 0~327680] snapc 14=[14,13,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-19T02:04:00.262 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:46.466925+0000", 2024-04-19T02:04:00.262 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 13.716760417, 2024-04-19T02:04:00.262 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014041065, 2024-04-19T02:04:00.262 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:00.263 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:00.263 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:00.263 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:04:00.263 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:04:00.263 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:00.263 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 700 2024-04-19T02:04:00.263 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.263 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:00.263 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.263 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:00.263 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.466925+0000", 2024-04-19T02:04:00.263 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.263 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.263 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.263 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:00.264 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.466925+0000", 2024-04-19T02:04:00.264 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.264 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.264 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.264 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:00.264 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.466928+0000", 2024-04-19T02:04:00.264 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6340000000000001e-06 2024-04-19T02:04:00.264 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.264 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.264 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:00.264 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.467539+0000", 2024-04-19T02:04:00.264 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00061067499999999998 2024-04-19T02:04:00.264 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.264 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.264 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:00.265 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.467570+0000", 2024-04-19T02:04:00.265 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1049e-05 2024-04-19T02:04:00.265 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.265 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.265 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.265 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.467597+0000", 2024-04-19T02:04:00.265 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7042e-05 2024-04-19T02:04:00.265 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.265 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.265 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:00.265 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.470887+0000", 2024-04-19T02:04:00.265 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0032905349999999998 2024-04-19T02:04:00.265 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.265 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.265 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:04:00.265 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.470985+0000", 2024-04-19T02:04:00.266 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.8163000000000002e-05 2024-04-19T02:04:00.266 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.266 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.266 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:00.266 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.471797+0000", 2024-04-19T02:04:00.266 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00081176200000000005 2024-04-19T02:04:00.266 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.266 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.266 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:00.266 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.480293+0000", 2024-04-19T02:04:00.266 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0084961099999999994 2024-04-19T02:04:00.266 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.266 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.267 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:04:00.267 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.480852+0000", 2024-04-19T02:04:00.267 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00055856000000000005 2024-04-19T02:04:00.267 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.267 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.267 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:00.267 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.480966+0000", 2024-04-19T02:04:00.267 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000114535 2024-04-19T02:04:00.267 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.267 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:00.267 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.267 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.267 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.267 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:701 3.es0 3:79fe7502:::smithi05930959-12:head [setxattr _header (57) in=7b] snapc 14=[14,13,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-19T02:04:00.267 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:46.467631+0000", 2024-04-19T02:04:00.268 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 13.716054691, 2024-04-19T02:04:00.268 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.016629863000000002, 2024-04-19T02:04:00.268 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:00.268 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:00.268 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:00.268 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:04:00.268 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:04:00.268 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:00.268 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 701 2024-04-19T02:04:00.268 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.268 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:00.268 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.268 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:00.268 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.467631+0000", 2024-04-19T02:04:00.268 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.269 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.269 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.269 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:00.269 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.467631+0000", 2024-04-19T02:04:00.269 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.269 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.269 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.269 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:00.269 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.467633+0000", 2024-04-19T02:04:00.269 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9e-06 2024-04-19T02:04:00.269 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.269 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.269 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:00.269 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.467643+0000", 2024-04-19T02:04:00.269 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0626e-05 2024-04-19T02:04:00.270 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.270 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.270 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:00.270 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.467663+0000", 2024-04-19T02:04:00.270 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9765000000000001e-05 2024-04-19T02:04:00.270 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.270 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.270 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.270 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.467681+0000", 2024-04-19T02:04:00.270 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7790000000000001e-05 2024-04-19T02:04:00.270 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.270 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.270 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:00.270 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.474847+0000", 2024-04-19T02:04:00.271 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.007165902 2024-04-19T02:04:00.271 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.271 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.271 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:04:00.271 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.475123+0000", 2024-04-19T02:04:00.271 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00027593099999999998 2024-04-19T02:04:00.271 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.271 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.271 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:00.271 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.476448+0000", 2024-04-19T02:04:00.271 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0013249990000000001 2024-04-19T02:04:00.271 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.271 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.271 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:00.272 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.483111+0000", 2024-04-19T02:04:00.272 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0066628269999999996 2024-04-19T02:04:00.272 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.272 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.272 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:04:00.272 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.483262+0000", 2024-04-19T02:04:00.272 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015127900000000001 2024-04-19T02:04:00.272 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.272 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.272 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:00.272 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.484261+0000", 2024-04-19T02:04:00.272 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00099884400000000003 2024-04-19T02:04:00.272 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.272 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:00.272 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.273 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.273 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.273 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:702 3.es0 3:79fe7502:::smithi05930959-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-19T02:04:00.273 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:46.467702+0000", 2024-04-19T02:04:00.273 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 13.715983311, 2024-04-19T02:04:00.273 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.018706202000000002, 2024-04-19T02:04:00.273 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:00.273 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:00.273 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:04:00.273 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:04:00.273 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:04:00.273 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:00.273 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 702 2024-04-19T02:04:00.273 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.274 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:00.274 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.274 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:00.274 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.467702+0000", 2024-04-19T02:04:00.274 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.274 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.274 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.274 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:00.274 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.467702+0000", 2024-04-19T02:04:00.274 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:00.274 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.274 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.274 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:00.275 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.467704+0000", 2024-04-19T02:04:00.275 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.257e-06 2024-04-19T02:04:00.275 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.275 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.275 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:00.275 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.467710+0000", 2024-04-19T02:04:00.275 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.263e-06 2024-04-19T02:04:00.275 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.275 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.275 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:00.275 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.467734+0000", 2024-04-19T02:04:00.275 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4252000000000001e-05 2024-04-19T02:04:00.275 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.275 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.276 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.276 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.467748+0000", 2024-04-19T02:04:00.276 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3686000000000001e-05 2024-04-19T02:04:00.276 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.276 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.276 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:00.276 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.478572+0000", 2024-04-19T02:04:00.276 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010824088000000001 2024-04-19T02:04:00.276 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.276 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.276 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for rw locks", 2024-04-19T02:04:00.276 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.478632+0000", 2024-04-19T02:04:00.276 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0044999999999999e-05 2024-04-19T02:04:00.276 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.277 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.277 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:00.277 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.484322+0000", 2024-04-19T02:04:00.277 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0056901590000000002 2024-04-19T02:04:00.277 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.277 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.277 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:04:00.277 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.484425+0000", 2024-04-19T02:04:00.277 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010281900000000001 2024-04-19T02:04:00.277 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:00.277 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:00.277 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:00.277 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:46.486409+0000", 2024-04-19T02:04:00.277 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0019836329999999998 2024-04-19T02:04:00.278 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.278 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:00.278 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.278 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:00.278 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:00.278 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:04:00.279 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 43 v 67 (ObjNum 33 snap 0 seq_num 1243827472) dirty exists 2024-04-19T02:04:00.279 INFO:tasks.rados.rados.0.smithi059.stdout:335: finishing rollback tid 1 to smithi05930959-26 2024-04-19T02:04:00.279 INFO:tasks.rados.rados.0.smithi059.stdout:341: finishing rollback tid 0 to smithi05930959-32 2024-04-19T02:04:00.280 INFO:tasks.rados.rados.0.smithi059.stdout:334: finishing write tid 1 to smithi05930959-49 2024-04-19T02:04:00.280 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 36 v 35 (ObjNum 147 snap 25 seq_num 147) dirty exists 2024-04-19T02:04:00.280 INFO:tasks.rados.rados.0.smithi059.stdout:335: finishing rollback tid 2 to smithi05930959-26 2024-04-19T02:04:00.280 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 26 v 101 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-19T02:04:00.280 INFO:tasks.rados.rados.0.smithi059.stdout:334: finishing write tid 2 to smithi05930959-49 2024-04-19T02:04:00.280 INFO:tasks.rados.rados.0.smithi059.stdout:341: finishing rollback tid 1 to smithi05930959-32 2024-04-19T02:04:00.280 INFO:tasks.rados.rados.0.smithi059.stdout:341: finishing rollback tid 2 to smithi05930959-32 2024-04-19T02:04:00.280 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 32 v 155 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-19T02:04:00.280 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 10 v 156 (ObjNum 88 snap 9 seq_num 88) dirty exists 2024-04-19T02:04:00.280 INFO:tasks.rados.rados.0.smithi059.stdout:334: finishing write tid 3 to smithi05930959-49 2024-04-19T02:04:00.280 INFO:tasks.rados.rados.0.smithi059.stdout:334: oid 49 updating version 0 to 46 2024-04-19T02:04:00.280 INFO:tasks.rados.rados.0.smithi059.stdout:334: oid 49 version 46 is already newer than 45 2024-04-19T02:04:00.280 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 49 v 46 (ObjNum 151 snap 28 seq_num 151) dirty exists 2024-04-19T02:04:00.280 INFO:tasks.rados.rados.0.smithi059.stdout:334: left oid 49 (ObjNum 151 snap 28 seq_num 151) 2024-04-19T02:04:00.280 INFO:tasks.rados.rados.0.smithi059.stdout:336: expect (ObjNum 129 snap 19 seq_num 129) 2024-04-19T02:04:00.281 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:04:00.281+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:04:00.281 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-19T02:04:00.318 INFO:tasks.rados.rados.0.smithi059.stdout:333: finishing copy_from to smithi05930959-40 2024-04-19T02:04:00.318 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 40 v 141 (ObjNum 142 snap 24 seq_num 142) dirty exists 2024-04-19T02:04:00.318 INFO:tasks.rados.rados.0.smithi059.stdout:343: expect (ObjNum 117 snap 17 seq_num 117) 2024-04-19T02:04:00.380 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:04:00.381+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:04:00.411 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:04:00.411 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:04:00.411 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-04-19T02:04:00.411 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:04:00.424 INFO:tasks.rados.rados.0.smithi059.stdout:345: finishing write tid 1 to smithi05930959-4 2024-04-19T02:04:00.424 INFO:tasks.rados.rados.0.smithi059.stdout:345: finishing write tid 2 to smithi05930959-4 2024-04-19T02:04:00.424 INFO:tasks.rados.rados.0.smithi059.stdout:345: finishing write tid 3 to smithi05930959-4 2024-04-19T02:04:00.424 INFO:tasks.rados.rados.0.smithi059.stdout:345: oid 4 updating version 0 to 92 2024-04-19T02:04:00.424 INFO:tasks.rados.rados.0.smithi059.stdout:345: oid 4 version 92 is already newer than 91 2024-04-19T02:04:00.424 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 4 v 92 (ObjNum 152 snap 28 seq_num 152) dirty exists 2024-04-19T02:04:00.424 INFO:tasks.rados.rados.0.smithi059.stdout:345: left oid 4 (ObjNum 152 snap 28 seq_num 152) 2024-04-19T02:04:00.424 INFO:tasks.rados.rados.0.smithi059.stdout:346: expect (ObjNum 91 snap 9 seq_num 91) 2024-04-19T02:04:00.425 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-19T02:04:00.481 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:04:00.481+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:04:00.523 INFO:tasks.rados.rados.0.smithi059.stdout:338: finishing copy_from racing read to smithi05930959-41 2024-04-19T02:04:00.523 INFO:tasks.rados.rados.0.smithi059.stdout:338: finishing copy_from to smithi05930959-41 2024-04-19T02:04:00.524 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 41 v 73 (ObjNum 136 snap 21 seq_num 136) dirty exists 2024-04-19T02:04:00.542 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:04:00.542 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:04:00.542 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-04-19T02:04:00.543 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-04-19T02:04:00.543 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:04:00.556 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-19T02:04:00.582 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:04:00.581+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:04:00.683 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:04:00.685+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:04:00.723 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:04:00.723 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-04-19T02:04:00.723 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-04-19T02:04:00.723 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-04-19T02:04:00.723 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.723 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:04:00.724 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:02:59.539411+0000", 2024-04-19T02:04:00.724 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 61.183702683999996, 2024-04-19T02:04:00.724 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.43600685099999997, 2024-04-19T02:04:00.724 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:00.724 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:00.724 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:00.724 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:00.724 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:04:00.724 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:04:00.724 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 1 2024-04-19T02:04:00.724 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.724 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:00.724 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.725 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:00.725 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539411+0000", 2024-04-19T02:04:00.725 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.725 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.725 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.725 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:00.725 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539411+0000", 2024-04-19T02:04:00.725 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.725 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.725 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.725 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:00.725 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539413+0000", 2024-04-19T02:04:00.725 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3970000000000001e-06 2024-04-19T02:04:00.725 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.725 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.725 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:00.726 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539429+0000", 2024-04-19T02:04:00.726 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5153e-05 2024-04-19T02:04:00.726 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.726 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.726 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:00.726 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539450+0000", 2024-04-19T02:04:00.726 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1243e-05 2024-04-19T02:04:00.726 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.726 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.726 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.726 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539470+0000", 2024-04-19T02:04:00.726 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0017000000000002e-05 2024-04-19T02:04:00.727 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.727 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.727 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:00.727 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539605+0000", 2024-04-19T02:04:00.727 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013489200000000001 2024-04-19T02:04:00.727 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.727 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.727 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-04-19T02:04:00.727 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.539633+0000", 2024-04-19T02:04:00.727 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8113e-05 2024-04-19T02:04:00.727 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.727 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.727 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:00.727 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.972415+0000", 2024-04-19T02:04:00.727 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.43278235300000001 2024-04-19T02:04:00.728 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.728 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.728 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:00.728 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.972942+0000", 2024-04-19T02:04:00.728 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000526379 2024-04-19T02:04:00.728 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.728 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.728 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:04:00.728 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.973384+0000", 2024-04-19T02:04:00.728 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00044265 2024-04-19T02:04:00.728 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.728 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.728 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:04:00.728 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.974795+0000", 2024-04-19T02:04:00.728 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001410592 2024-04-19T02:04:00.729 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.729 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.729 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:04:00.729 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.975249+0000", 2024-04-19T02:04:00.729 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00045376999999999999 2024-04-19T02:04:00.729 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.729 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.729 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:04:00.729 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.975346+0000", 2024-04-19T02:04:00.729 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7916000000000002e-05 2024-04-19T02:04:00.729 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.729 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.729 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:00.729 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:02:59.975418+0000", 2024-04-19T02:04:00.729 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1376000000000002e-05 2024-04-19T02:04:00.730 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.730 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:00.730 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.730 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.730 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.730 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:04:00.730 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.791660+0000", 2024-04-19T02:04:00.730 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 59.931453310999999, 2024-04-19T02:04:00.730 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021026688000000002, 2024-04-19T02:04:00.730 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:00.731 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:00.731 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:00.731 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:00.731 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:04:00.731 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:04:00.731 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 139 2024-04-19T02:04:00.731 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.731 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:00.731 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.731 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:00.731 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791660+0000", 2024-04-19T02:04:00.731 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.731 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.732 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.732 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:00.732 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791660+0000", 2024-04-19T02:04:00.732 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.732 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.732 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.732 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:00.732 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791661+0000", 2024-04-19T02:04:00.732 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2699999999999999e-07 2024-04-19T02:04:00.732 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.732 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.732 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:00.732 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791666+0000", 2024-04-19T02:04:00.733 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4160000000000003e-06 2024-04-19T02:04:00.733 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.733 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.733 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:00.733 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791676+0000", 2024-04-19T02:04:00.733 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1169999999999993e-06 2024-04-19T02:04:00.733 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.733 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.733 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.733 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791683+0000", 2024-04-19T02:04:00.733 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8369999999999997e-06 2024-04-19T02:04:00.733 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.733 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.733 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:00.733 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.807160+0000", 2024-04-19T02:04:00.734 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.01547656 2024-04-19T02:04:00.734 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.734 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.734 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:00.734 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.807408+0000", 2024-04-19T02:04:00.734 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000248485 2024-04-19T02:04:00.734 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.734 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.734 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:04:00.734 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.807622+0000", 2024-04-19T02:04:00.734 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021381199999999999 2024-04-19T02:04:00.734 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.734 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.734 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:04:00.735 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812397+0000", 2024-04-19T02:04:00.735 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0047750229999999998 2024-04-19T02:04:00.735 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.735 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.735 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:04:00.735 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812434+0000", 2024-04-19T02:04:00.735 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6869000000000002e-05 2024-04-19T02:04:00.735 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.735 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.735 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:04:00.735 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812571+0000", 2024-04-19T02:04:00.735 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013649199999999999 2024-04-19T02:04:00.735 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.735 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.735 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:00.736 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812687+0000", 2024-04-19T02:04:00.736 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011635 2024-04-19T02:04:00.736 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.736 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:00.736 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.736 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.736 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.736 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:04:00.736 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.791694+0000", 2024-04-19T02:04:00.736 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 59.931419390999999, 2024-04-19T02:04:00.736 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021303507999999999, 2024-04-19T02:04:00.736 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:00.736 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:00.737 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:00.737 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:00.737 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:04:00.737 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:04:00.737 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 140 2024-04-19T02:04:00.737 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.737 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:00.737 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.737 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:00.737 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791694+0000", 2024-04-19T02:04:00.737 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.737 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.737 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.737 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:00.738 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791694+0000", 2024-04-19T02:04:00.738 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.738 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.738 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.738 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:00.738 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791695+0000", 2024-04-19T02:04:00.738 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2699999999999999e-07 2024-04-19T02:04:00.738 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.738 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.738 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:00.738 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791829+0000", 2024-04-19T02:04:00.738 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013436500000000001 2024-04-19T02:04:00.738 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.738 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.739 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:00.739 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791841+0000", 2024-04-19T02:04:00.739 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1707e-05 2024-04-19T02:04:00.739 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.739 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.739 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.739 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791850+0000", 2024-04-19T02:04:00.739 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5799999999999992e-06 2024-04-19T02:04:00.739 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.739 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.739 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:00.739 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.808635+0000", 2024-04-19T02:04:00.739 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016785133000000001 2024-04-19T02:04:00.739 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.740 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.740 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:00.740 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.808862+0000", 2024-04-19T02:04:00.740 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022693799999999999 2024-04-19T02:04:00.740 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.740 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.740 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:04:00.740 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.809242+0000", 2024-04-19T02:04:00.740 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00038053400000000003 2024-04-19T02:04:00.740 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.740 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.740 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:04:00.740 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812709+0000", 2024-04-19T02:04:00.740 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0034666929999999999 2024-04-19T02:04:00.740 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.741 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.741 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:04:00.741 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812796+0000", 2024-04-19T02:04:00.741 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6877000000000002e-05 2024-04-19T02:04:00.741 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.741 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.741 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:04:00.741 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812910+0000", 2024-04-19T02:04:00.741 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000113929 2024-04-19T02:04:00.741 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.741 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.741 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:00.741 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.812998+0000", 2024-04-19T02:04:00.741 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8024999999999998e-05 2024-04-19T02:04:00.742 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.742 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:00.742 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.742 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.742 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.742 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4207.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-04-19T02:04:00.742 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:00.791863+0000", 2024-04-19T02:04:00.742 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 59.931250581999997, 2024-04-19T02:04:00.742 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021620092, 2024-04-19T02:04:00.742 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:00.742 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:00.742 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:00.742 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:00.742 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4207", 2024-04-19T02:04:00.743 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.81:0/2157977180", 2024-04-19T02:04:00.743 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 141 2024-04-19T02:04:00.743 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.743 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:00.743 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.743 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:00.743 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791863+0000", 2024-04-19T02:04:00.743 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.743 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.743 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.743 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:00.743 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791863+0000", 2024-04-19T02:04:00.743 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.743 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.744 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.744 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:00.744 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791864+0000", 2024-04-19T02:04:00.744 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0599999999999999e-07 2024-04-19T02:04:00.744 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.744 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.744 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:00.744 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791871+0000", 2024-04-19T02:04:00.744 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1040000000000001e-06 2024-04-19T02:04:00.744 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.744 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.744 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:00.744 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791881+0000", 2024-04-19T02:04:00.744 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4329999999999993e-06 2024-04-19T02:04:00.745 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.745 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.745 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.745 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.791888+0000", 2024-04-19T02:04:00.745 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7360000000000001e-06 2024-04-19T02:04:00.745 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.745 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.745 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:00.745 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.811476+0000", 2024-04-19T02:04:00.745 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019588186 2024-04-19T02:04:00.745 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.745 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.745 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:00.745 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.811595+0000", 2024-04-19T02:04:00.746 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011901200000000001 2024-04-19T02:04:00.746 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.746 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.746 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-04-19T02:04:00.746 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.811793+0000", 2024-04-19T02:04:00.746 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000197958 2024-04-19T02:04:00.746 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.746 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.746 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-04-19T02:04:00.746 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.813066+0000", 2024-04-19T02:04:00.746 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001272884 2024-04-19T02:04:00.746 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.746 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.746 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-04-19T02:04:00.747 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.813356+0000", 2024-04-19T02:04:00.747 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028986399999999997 2024-04-19T02:04:00.747 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.747 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.747 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:04:00.747 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.813430+0000", 2024-04-19T02:04:00.747 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3716999999999999e-05 2024-04-19T02:04:00.747 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.747 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.747 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:00.747 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:00.813483+0000", 2024-04-19T02:04:00.747 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3291999999999997e-05 2024-04-19T02:04:00.747 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.747 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:00.747 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.748 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.748 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.748 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:363 3.ds0 3:bdf44503:::smithi05930959-38:head [append 0~638976] snapc 9=[9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:04:00.748 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.746432+0000", 2024-04-19T02:04:00.748 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 33.976682126999997, 2024-04-19T02:04:00.748 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.037983646000000003, 2024-04-19T02:04:00.748 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:00.748 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:00.748 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:00.748 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:00.748 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:00.748 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:00.748 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 363 2024-04-19T02:04:00.748 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.748 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:00.749 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.749 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:00.749 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.746432+0000", 2024-04-19T02:04:00.749 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.749 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.749 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.749 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:00.749 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.746432+0000", 2024-04-19T02:04:00.749 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.749 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.749 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.749 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:00.749 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.746435+0000", 2024-04-19T02:04:00.749 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3160000000000001e-06 2024-04-19T02:04:00.750 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.750 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.750 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:00.750 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747630+0000", 2024-04-19T02:04:00.750 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001195371 2024-04-19T02:04:00.750 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.750 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.750 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:00.750 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747664+0000", 2024-04-19T02:04:00.750 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4190000000000003e-05 2024-04-19T02:04:00.750 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.750 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.750 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.751 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747687+0000", 2024-04-19T02:04:00.751 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2368e-05 2024-04-19T02:04:00.751 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.751 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.751 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:00.751 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.747968+0000", 2024-04-19T02:04:00.751 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028139799999999998 2024-04-19T02:04:00.751 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.751 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.751 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:00.751 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.748247+0000", 2024-04-19T02:04:00.751 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00027916600000000002 2024-04-19T02:04:00.751 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.751 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.751 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:00.752 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.753590+0000", 2024-04-19T02:04:00.752 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0053429920000000004 2024-04-19T02:04:00.752 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.752 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.752 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:00.752 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.782450+0000", 2024-04-19T02:04:00.752 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.028859510000000001 2024-04-19T02:04:00.752 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.752 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.752 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:04:00.752 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.784240+0000", 2024-04-19T02:04:00.752 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001790224 2024-04-19T02:04:00.752 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.752 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.753 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:00.753 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.784415+0000", 2024-04-19T02:04:00.753 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017511100000000001 2024-04-19T02:04:00.753 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.753 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:00.753 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.753 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.753 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.753 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.5s4 42/26 ECSubWrite(tid=51, reqid=osd.10.0:51, at_version=42'27, trim_to=0'0, roll_forward_to=39'25))", 2024-04-19T02:04:00.753 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.754979+0000", 2024-04-19T02:04:00.753 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 33.968134474000003, 2024-04-19T02:04:00.753 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.027548537000000001, 2024-04-19T02:04:00.753 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:00.753 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:00.753 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:04:00.754 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:00.754 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.754 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:00.754 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.754979+0000", 2024-04-19T02:04:00.754 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.754 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.754 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.754 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:00.754 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.754979+0000", 2024-04-19T02:04:00.754 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.754 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.754 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.754 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:00.754 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.754980+0000", 2024-04-19T02:04:00.755 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9999999999999996e-07 2024-04-19T02:04:00.755 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.755 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.755 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:00.755 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755050+0000", 2024-04-19T02:04:00.755 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9447000000000001e-05 2024-04-19T02:04:00.755 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.755 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.755 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:00.755 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755077+0000", 2024-04-19T02:04:00.755 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7948000000000001e-05 2024-04-19T02:04:00.755 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.755 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.755 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.755 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755100+0000", 2024-04-19T02:04:00.756 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2714999999999999e-05 2024-04-19T02:04:00.756 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.756 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.756 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:00.756 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760119+0000", 2024-04-19T02:04:00.756 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0050189800000000001 2024-04-19T02:04:00.756 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.756 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.756 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:00.756 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760142+0000", 2024-04-19T02:04:00.756 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2603999999999999e-05 2024-04-19T02:04:00.756 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.756 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.756 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:00.757 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.782453+0000", 2024-04-19T02:04:00.757 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022311218000000001 2024-04-19T02:04:00.757 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.757 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.757 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:00.757 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.782528+0000", 2024-04-19T02:04:00.757 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4725e-05 2024-04-19T02:04:00.757 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.757 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:00.757 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.757 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.757 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.757 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:364 3.ds0 3:bdf44503:::smithi05930959-38:head [append 0~622592] snapc 9=[9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:04:00.757 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.756160+0000", 2024-04-19T02:04:00.758 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 33.966954133000002, 2024-04-19T02:04:00.758 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.034510458000000001, 2024-04-19T02:04:00.758 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:00.758 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:00.758 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:00.758 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:00.758 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:00.758 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:00.758 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 364 2024-04-19T02:04:00.758 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.758 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:00.758 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.758 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:00.758 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.756160+0000", 2024-04-19T02:04:00.759 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.759 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.759 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.759 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:00.759 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.756160+0000", 2024-04-19T02:04:00.759 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.759 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.759 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.759 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:00.759 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.756163+0000", 2024-04-19T02:04:00.759 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5130000000000001e-06 2024-04-19T02:04:00.759 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.759 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.759 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:00.760 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.757299+0000", 2024-04-19T02:04:00.760 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0011355289999999999 2024-04-19T02:04:00.760 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.760 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.760 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:00.760 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.757340+0000", 2024-04-19T02:04:00.760 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1680000000000001e-05 2024-04-19T02:04:00.760 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.760 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.760 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.760 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.757364+0000", 2024-04-19T02:04:00.760 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4215000000000001e-05 2024-04-19T02:04:00.760 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.760 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.760 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:00.760 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.763302+0000", 2024-04-19T02:04:00.761 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0059373480000000003 2024-04-19T02:04:00.761 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.761 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.761 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:00.761 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.768600+0000", 2024-04-19T02:04:00.761 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.005298568 2024-04-19T02:04:00.761 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.761 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.761 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:00.761 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.772526+0000", 2024-04-19T02:04:00.761 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039256450000000002 2024-04-19T02:04:00.761 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.761 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.761 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:00.762 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.790307+0000", 2024-04-19T02:04:00.762 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017781165000000002 2024-04-19T02:04:00.762 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.762 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.762 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:04:00.762 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.790491+0000", 2024-04-19T02:04:00.762 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000183845 2024-04-19T02:04:00.762 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.762 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.762 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:00.762 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.790670+0000", 2024-04-19T02:04:00.762 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017895000000000001 2024-04-19T02:04:00.762 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.762 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:00.763 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.763 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.763 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.763 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:365 3.ds0 3:bdf44503:::smithi05930959-38:head [append 0~147456] snapc 9=[9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:04:00.763 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.758315+0000", 2024-04-19T02:04:00.763 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 33.964798956999999, 2024-04-19T02:04:00.763 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.040637806999999998, 2024-04-19T02:04:00.763 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:00.763 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:00.763 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:00.763 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:00.763 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:00.763 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:00.763 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 365 2024-04-19T02:04:00.764 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.764 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:00.764 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.764 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:00.764 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758315+0000", 2024-04-19T02:04:00.764 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.764 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.764 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.764 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:00.764 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758315+0000", 2024-04-19T02:04:00.764 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.764 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.764 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.764 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:00.764 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758317+0000", 2024-04-19T02:04:00.765 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5129999999999999e-06 2024-04-19T02:04:00.765 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.765 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.765 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:00.765 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758592+0000", 2024-04-19T02:04:00.765 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000275285 2024-04-19T02:04:00.765 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.765 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.765 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:00.765 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758625+0000", 2024-04-19T02:04:00.765 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2668e-05 2024-04-19T02:04:00.765 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.765 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.766 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.766 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758662+0000", 2024-04-19T02:04:00.766 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7042999999999998e-05 2024-04-19T02:04:00.766 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.766 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.766 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:00.766 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.784642+0000", 2024-04-19T02:04:00.766 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.025979510000000001 2024-04-19T02:04:00.766 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.766 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.766 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:00.766 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.785163+0000", 2024-04-19T02:04:00.766 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00052105999999999995 2024-04-19T02:04:00.766 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.767 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.767 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:00.767 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.788827+0000", 2024-04-19T02:04:00.767 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0036638109999999999 2024-04-19T02:04:00.767 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.767 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.767 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:00.767 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.792982+0000", 2024-04-19T02:04:00.767 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0041554829999999997 2024-04-19T02:04:00.767 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.767 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.767 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:04:00.767 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.798798+0000", 2024-04-19T02:04:00.767 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0058162379999999996 2024-04-19T02:04:00.768 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.768 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.768 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:00.768 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.798952+0000", 2024-04-19T02:04:00.768 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000154196 2024-04-19T02:04:00.768 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.768 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:00.768 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.768 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.768 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.768 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:366 3.ds0 3:bdf44503:::smithi05930959-38:head [setxattr _header (56) in=7b] snapc 9=[9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:04:00.768 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.758698+0000", 2024-04-19T02:04:00.768 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 33.964416030000002, 2024-04-19T02:04:00.769 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.041876360000000001, 2024-04-19T02:04:00.769 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:00.769 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:00.769 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:00.769 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:00.769 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:00.769 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:00.769 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 366 2024-04-19T02:04:00.769 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.769 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:00.769 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.769 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:00.769 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758698+0000", 2024-04-19T02:04:00.769 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.769 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.770 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.770 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:00.770 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758698+0000", 2024-04-19T02:04:00.770 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.770 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.770 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.770 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:00.770 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758699+0000", 2024-04-19T02:04:00.770 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8160000000000001e-06 2024-04-19T02:04:00.770 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.770 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.770 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:00.770 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758711+0000", 2024-04-19T02:04:00.770 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1070000000000001e-05 2024-04-19T02:04:00.771 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.771 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.771 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:00.771 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758731+0000", 2024-04-19T02:04:00.771 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0650999999999999e-05 2024-04-19T02:04:00.771 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.771 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.771 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.771 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758747+0000", 2024-04-19T02:04:00.771 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5452e-05 2024-04-19T02:04:00.771 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.771 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.771 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:00.771 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.790875+0000", 2024-04-19T02:04:00.771 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.032128139999999999 2024-04-19T02:04:00.772 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.772 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.772 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:00.772 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.791060+0000", 2024-04-19T02:04:00.772 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00018499200000000001 2024-04-19T02:04:00.772 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.772 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.772 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:00.772 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.791992+0000", 2024-04-19T02:04:00.772 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00093232600000000003 2024-04-19T02:04:00.772 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.772 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.772 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:00.772 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.796307+0000", 2024-04-19T02:04:00.773 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0043147899999999998 2024-04-19T02:04:00.773 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.773 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.773 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:04:00.773 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.800345+0000", 2024-04-19T02:04:00.773 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.004037789 2024-04-19T02:04:00.773 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.773 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.773 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:00.773 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.800574+0000", 2024-04-19T02:04:00.773 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022933400000000001 2024-04-19T02:04:00.773 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.773 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:00.773 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.773 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.774 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.774 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:367 3.ds0 3:bdf44503:::smithi05930959-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:04:00.774 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.758769+0000", 2024-04-19T02:04:00.774 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 33.964344965999999, 2024-04-19T02:04:00.774 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.049021666999999998, 2024-04-19T02:04:00.774 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:00.774 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:00.774 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:04:00.774 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:00.774 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:00.774 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:00.774 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 367 2024-04-19T02:04:00.774 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.774 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:00.774 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.775 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:00.775 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758769+0000", 2024-04-19T02:04:00.775 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.775 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.775 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.775 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:00.775 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758769+0000", 2024-04-19T02:04:00.775 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.775 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.775 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.775 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:00.775 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758770+0000", 2024-04-19T02:04:00.775 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5829999999999999e-06 2024-04-19T02:04:00.775 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.775 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.775 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:00.776 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758794+0000", 2024-04-19T02:04:00.776 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3408e-05 2024-04-19T02:04:00.776 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.776 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.776 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:00.776 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758815+0000", 2024-04-19T02:04:00.776 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1049000000000001e-05 2024-04-19T02:04:00.776 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.776 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.776 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.776 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.758830+0000", 2024-04-19T02:04:00.776 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4975e-05 2024-04-19T02:04:00.776 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.776 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.776 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:00.776 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.793116+0000", 2024-04-19T02:04:00.776 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.034285806000000002 2024-04-19T02:04:00.777 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.777 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.777 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:04:00.777 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.793286+0000", 2024-04-19T02:04:00.777 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017076099999999999 2024-04-19T02:04:00.777 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.777 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.777 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:00.777 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.801267+0000", 2024-04-19T02:04:00.777 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0079808080000000007 2024-04-19T02:04:00.777 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.777 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.777 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:00.777 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.801431+0000", 2024-04-19T02:04:00.777 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016360499999999999 2024-04-19T02:04:00.778 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.778 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.778 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:00.778 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.807790+0000", 2024-04-19T02:04:00.778 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0063596720000000002 2024-04-19T02:04:00.778 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.778 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:00.778 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.778 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.778 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.778 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:692 3.ds0 3:b3d82b95:::smithi05930959-4:head [append 0~622592] snapc 14=[14,13,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-19T02:04:00.778 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:46.447436+0000", 2024-04-19T02:04:00.778 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 14.275677845000001, 2024-04-19T02:04:00.778 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.025270186, 2024-04-19T02:04:00.778 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:00.779 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:00.779 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:00.779 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:00.779 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:00.779 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:00.779 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 692 2024-04-19T02:04:00.779 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.779 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:00.779 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.779 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:00.779 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.447436+0000", 2024-04-19T02:04:00.779 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.779 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.779 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.779 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:00.780 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.447436+0000", 2024-04-19T02:04:00.780 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.780 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.780 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.780 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:00.780 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.447438+0000", 2024-04-19T02:04:00.780 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3769999999999999e-06 2024-04-19T02:04:00.780 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.780 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.780 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:00.780 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.448282+0000", 2024-04-19T02:04:00.780 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00084384900000000003 2024-04-19T02:04:00.780 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.780 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.780 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:00.781 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.448312+0000", 2024-04-19T02:04:00.781 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9975e-05 2024-04-19T02:04:00.781 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.781 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.781 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.781 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.448333+0000", 2024-04-19T02:04:00.781 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1049000000000001e-05 2024-04-19T02:04:00.781 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.781 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.781 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:00.781 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.453182+0000", 2024-04-19T02:04:00.781 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0048494230000000003 2024-04-19T02:04:00.781 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.781 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.781 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:00.781 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.453430+0000", 2024-04-19T02:04:00.782 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00024780899999999999 2024-04-19T02:04:00.782 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.782 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.782 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:00.782 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.455753+0000", 2024-04-19T02:04:00.782 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0023230999999999998 2024-04-19T02:04:00.782 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.782 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.782 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:00.782 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.472262+0000", 2024-04-19T02:04:00.782 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016509113999999998 2024-04-19T02:04:00.782 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.782 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.782 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:04:00.783 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.472489+0000", 2024-04-19T02:04:00.783 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022607000000000001 2024-04-19T02:04:00.783 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.783 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.783 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:00.783 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.472706+0000", 2024-04-19T02:04:00.783 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021741999999999999 2024-04-19T02:04:00.783 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.783 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:00.783 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.783 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.783 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.783 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:693 3.ds0 3:b3d82b95:::smithi05930959-4:head [append 0~622592] snapc 14=[14,13,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-19T02:04:00.783 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:46.451588+0000", 2024-04-19T02:04:00.784 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 14.271525941, 2024-04-19T02:04:00.784 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022143603000000001, 2024-04-19T02:04:00.784 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:00.784 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:00.784 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:00.784 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:00.784 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:00.784 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:00.784 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 693 2024-04-19T02:04:00.784 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.784 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:00.784 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.784 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:00.784 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.451588+0000", 2024-04-19T02:04:00.785 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.785 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.785 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.785 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:00.785 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.451588+0000", 2024-04-19T02:04:00.785 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.785 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.785 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.785 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:00.785 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.451590+0000", 2024-04-19T02:04:00.785 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7240000000000001e-06 2024-04-19T02:04:00.785 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.785 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.785 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:00.785 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.452642+0000", 2024-04-19T02:04:00.786 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0010513160000000001 2024-04-19T02:04:00.786 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.786 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.786 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:00.786 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.452672+0000", 2024-04-19T02:04:00.786 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9762000000000001e-05 2024-04-19T02:04:00.786 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.786 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.786 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.786 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.452687+0000", 2024-04-19T02:04:00.786 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5469e-05 2024-04-19T02:04:00.786 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.786 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.786 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:00.787 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.459492+0000", 2024-04-19T02:04:00.787 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0068048099999999997 2024-04-19T02:04:00.787 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.787 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.787 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:00.787 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.459822+0000", 2024-04-19T02:04:00.787 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00032999800000000002 2024-04-19T02:04:00.787 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.787 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.787 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:00.788 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.462024+0000", 2024-04-19T02:04:00.788 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0022026319999999999 2024-04-19T02:04:00.788 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.788 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.788 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:00.788 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.472726+0000", 2024-04-19T02:04:00.788 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010701436999999999 2024-04-19T02:04:00.788 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.788 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.788 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:04:00.788 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.473503+0000", 2024-04-19T02:04:00.788 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00077692600000000005 2024-04-19T02:04:00.788 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.788 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.788 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:00.789 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.473731+0000", 2024-04-19T02:04:00.789 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022852899999999999 2024-04-19T02:04:00.789 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.789 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:00.789 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.789 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.789 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.789 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:694 3.ds0 3:b3d82b95:::smithi05930959-4:head [append 0~311296] snapc 14=[14,13,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-19T02:04:00.789 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:46.453801+0000", 2024-04-19T02:04:00.789 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 14.269312792999999, 2024-04-19T02:04:00.790 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.023717372, 2024-04-19T02:04:00.790 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:00.790 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:00.790 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:00.790 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:00.790 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:00.790 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:00.790 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 694 2024-04-19T02:04:00.790 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.790 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:00.790 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.790 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:00.790 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.453801+0000", 2024-04-19T02:04:00.791 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.791 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.791 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.791 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:00.791 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.453801+0000", 2024-04-19T02:04:00.791 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.791 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.791 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.791 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:00.791 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.453803+0000", 2024-04-19T02:04:00.791 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5969999999999999e-06 2024-04-19T02:04:00.791 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.791 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.791 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:00.792 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.454378+0000", 2024-04-19T02:04:00.792 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00057492100000000005 2024-04-19T02:04:00.792 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.792 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.792 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:00.792 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.454408+0000", 2024-04-19T02:04:00.792 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0037999999999999e-05 2024-04-19T02:04:00.792 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.792 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.792 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.792 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.454424+0000", 2024-04-19T02:04:00.793 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5177000000000001e-05 2024-04-19T02:04:00.793 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.793 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.793 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:00.793 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.464961+0000", 2024-04-19T02:04:00.793 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010537187999999999 2024-04-19T02:04:00.793 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.793 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.793 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:00.793 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.465414+0000", 2024-04-19T02:04:00.793 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00045295999999999997 2024-04-19T02:04:00.793 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.793 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.794 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:00.794 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.467274+0000", 2024-04-19T02:04:00.794 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0018604170000000001 2024-04-19T02:04:00.794 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.794 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.794 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:00.794 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.472730+0000", 2024-04-19T02:04:00.794 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0054556880000000002 2024-04-19T02:04:00.794 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.794 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.794 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:04:00.794 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.477358+0000", 2024-04-19T02:04:00.794 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0046284840000000004 2024-04-19T02:04:00.794 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.795 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.795 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:00.795 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.477518+0000", 2024-04-19T02:04:00.795 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015990199999999999 2024-04-19T02:04:00.795 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.795 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:00.795 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.795 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.795 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.795 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:695 3.ds0 3:b3d82b95:::smithi05930959-4:head [setxattr _header (56) in=7b] snapc 14=[14,13,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-19T02:04:00.795 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:46.454454+0000", 2024-04-19T02:04:00.795 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 14.268659531999999, 2024-04-19T02:04:00.795 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.024694714999999999, 2024-04-19T02:04:00.795 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:00.796 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:00.796 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:00.796 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:00.796 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:00.796 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:00.796 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 695 2024-04-19T02:04:00.796 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.796 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:00.796 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.796 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:00.796 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.454454+0000", 2024-04-19T02:04:00.796 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.796 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.796 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.796 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:00.797 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.454454+0000", 2024-04-19T02:04:00.797 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.797 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.797 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.797 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:00.797 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.454455+0000", 2024-04-19T02:04:00.797 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2470000000000001e-06 2024-04-19T02:04:00.797 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.797 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.797 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:00.797 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.454463+0000", 2024-04-19T02:04:00.797 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6860000000000006e-06 2024-04-19T02:04:00.797 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.797 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.798 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:00.798 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.454477+0000", 2024-04-19T02:04:00.798 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4192999999999999e-05 2024-04-19T02:04:00.798 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.798 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.798 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.798 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.454486+0000", 2024-04-19T02:04:00.798 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2099999999999999e-06 2024-04-19T02:04:00.798 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.798 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.798 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:00.798 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.469372+0000", 2024-04-19T02:04:00.798 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014885116 2024-04-19T02:04:00.798 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.799 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.799 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:00.799 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.469684+0000", 2024-04-19T02:04:00.799 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00031224600000000002 2024-04-19T02:04:00.799 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.799 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.799 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:00.799 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.471030+0000", 2024-04-19T02:04:00.799 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0013460200000000001 2024-04-19T02:04:00.799 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.799 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.799 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:00.799 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.477113+0000", 2024-04-19T02:04:00.799 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0060836550000000003 2024-04-19T02:04:00.799 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.800 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.800 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:04:00.800 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.478950+0000", 2024-04-19T02:04:00.800 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001836538 2024-04-19T02:04:00.800 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.800 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.800 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:00.800 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.479149+0000", 2024-04-19T02:04:00.800 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00019880399999999999 2024-04-19T02:04:00.800 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.800 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:00.800 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.800 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.800 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.800 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:696 3.ds0 3:b3d82b95:::smithi05930959-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-19T02:04:00.801 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:46.454499+0000", 2024-04-19T02:04:00.801 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 14.268614510999999, 2024-04-19T02:04:00.801 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.029722386, 2024-04-19T02:04:00.801 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:00.801 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:00.801 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:04:00.801 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:00.801 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:00.801 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:00.801 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 696 2024-04-19T02:04:00.801 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.801 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:00.801 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.801 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:00.801 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.454499+0000", 2024-04-19T02:04:00.802 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.802 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.802 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.802 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:00.802 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.454499+0000", 2024-04-19T02:04:00.802 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.802 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.802 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.802 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:00.802 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.454500+0000", 2024-04-19T02:04:00.802 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2699999999999998e-07 2024-04-19T02:04:00.802 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.802 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.802 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:00.803 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.454504+0000", 2024-04-19T02:04:00.803 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0729999999999998e-06 2024-04-19T02:04:00.803 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.803 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.803 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:00.803 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.454518+0000", 2024-04-19T02:04:00.803 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3623e-05 2024-04-19T02:04:00.803 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.803 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.803 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.803 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.454527+0000", 2024-04-19T02:04:00.803 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8799999999999997e-06 2024-04-19T02:04:00.803 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.803 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.803 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:00.804 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.476941+0000", 2024-04-19T02:04:00.804 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.02241485 2024-04-19T02:04:00.804 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.804 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.804 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:04:00.804 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.477102+0000", 2024-04-19T02:04:00.804 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000160416 2024-04-19T02:04:00.804 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.804 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.804 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:00.804 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.480818+0000", 2024-04-19T02:04:00.804 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0037161540000000002 2024-04-19T02:04:00.804 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.804 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.805 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:00.805 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.481076+0000", 2024-04-19T02:04:00.805 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00025776800000000002 2024-04-19T02:04:00.805 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.805 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.805 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:00.805 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:46.484222+0000", 2024-04-19T02:04:00.805 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0031457949999999998 2024-04-19T02:04:00.805 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.805 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:00.805 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.805 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.805 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.806 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.5s4 67/26 ECSubWrite(tid=85, reqid=client.4509.0:765, at_version=67'41, trim_to=0'0, roll_forward_to=64'39))", 2024-04-19T02:04:00.806 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:51.098386+0000", 2024-04-19T02:04:00.806 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.6247280079999999, 2024-04-19T02:04:00.806 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.030421074999999999, 2024-04-19T02:04:00.806 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:00.806 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:00.806 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:04:00.806 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:00.806 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.806 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:00.806 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.098386+0000", 2024-04-19T02:04:00.806 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.806 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.807 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.807 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:00.807 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.098386+0000", 2024-04-19T02:04:00.807 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.807 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.807 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.807 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:00.807 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.098386+0000", 2024-04-19T02:04:00.807 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9599999999999999e-07 2024-04-19T02:04:00.807 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.807 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.807 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:00.807 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.098635+0000", 2024-04-19T02:04:00.807 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00024883700000000002 2024-04-19T02:04:00.807 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.807 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.808 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:00.808 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.098661+0000", 2024-04-19T02:04:00.808 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5672000000000001e-05 2024-04-19T02:04:00.808 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.808 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.808 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.808 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.098680+0000", 2024-04-19T02:04:00.808 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8819000000000001e-05 2024-04-19T02:04:00.808 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.808 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.808 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:00.808 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.098859+0000", 2024-04-19T02:04:00.808 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017975700000000001 2024-04-19T02:04:00.808 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.808 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.808 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:00.809 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.098911+0000", 2024-04-19T02:04:00.809 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1198e-05 2024-04-19T02:04:00.809 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.809 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.809 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:00.809 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.128723+0000", 2024-04-19T02:04:00.809 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.029811983 2024-04-19T02:04:00.809 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.809 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.809 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:00.809 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.128807+0000", 2024-04-19T02:04:00.809 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4112999999999999e-05 2024-04-19T02:04:00.809 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.809 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:00.809 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.810 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.810 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.810 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:771 3.ds0 3:b3d82b95:::smithi05930959-4:15 [read 0~3817472,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-04-19T02:04:00.810 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:51.102304+0000", 2024-04-19T02:04:00.810 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.6208098779999993, 2024-04-19T02:04:00.810 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021210348, 2024-04-19T02:04:00.810 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:00.810 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:00.810 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:04:00.810 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:00.810 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:00.810 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:00.810 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 771 2024-04-19T02:04:00.810 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.811 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:00.811 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.811 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:00.811 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102304+0000", 2024-04-19T02:04:00.811 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.811 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.811 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.811 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:00.811 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102304+0000", 2024-04-19T02:04:00.811 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.811 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.811 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.811 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:00.811 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102308+0000", 2024-04-19T02:04:00.812 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7869999999999999e-06 2024-04-19T02:04:00.812 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.812 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.812 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:00.812 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102324+0000", 2024-04-19T02:04:00.812 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6218999999999998e-05 2024-04-19T02:04:00.812 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.812 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.812 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:00.812 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102359+0000", 2024-04-19T02:04:00.812 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5216000000000003e-05 2024-04-19T02:04:00.812 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.812 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.812 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.813 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102381+0000", 2024-04-19T02:04:00.813 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2322000000000001e-05 2024-04-19T02:04:00.813 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.813 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.813 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:00.813 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102733+0000", 2024-04-19T02:04:00.813 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00035188599999999998 2024-04-19T02:04:00.813 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.813 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.813 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:00.813 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.103012+0000", 2024-04-19T02:04:00.813 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00027872599999999997 2024-04-19T02:04:00.813 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.813 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.813 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:00.813 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.123514+0000", 2024-04-19T02:04:00.814 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020502191999999999 2024-04-19T02:04:00.814 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.814 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:00.814 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.814 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.814 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.814 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:772 3.ds0 3:b3d82b95:::smithi05930959-4:15 [read 0~3817472,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-04-19T02:04:00.814 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:51.102440+0000", 2024-04-19T02:04:00.814 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.620673923, 2024-04-19T02:04:00.814 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.025806882, 2024-04-19T02:04:00.814 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:00.814 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:00.814 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:04:00.815 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:00.815 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:00.815 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:00.815 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 772 2024-04-19T02:04:00.815 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.815 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:00.815 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.815 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:00.815 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102440+0000", 2024-04-19T02:04:00.815 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.815 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.815 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.815 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:00.815 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102440+0000", 2024-04-19T02:04:00.815 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.816 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.816 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.816 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:00.816 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102441+0000", 2024-04-19T02:04:00.816 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5200000000000001e-06 2024-04-19T02:04:00.816 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.816 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.816 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:00.816 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102451+0000", 2024-04-19T02:04:00.816 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3030000000000005e-06 2024-04-19T02:04:00.816 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.816 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.816 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:00.816 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102473+0000", 2024-04-19T02:04:00.816 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2453e-05 2024-04-19T02:04:00.817 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.817 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.817 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.817 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102489+0000", 2024-04-19T02:04:00.817 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6449000000000001e-05 2024-04-19T02:04:00.817 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.817 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.817 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:00.817 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.104146+0000", 2024-04-19T02:04:00.817 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001656287 2024-04-19T02:04:00.817 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.817 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.817 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:00.817 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.104421+0000", 2024-04-19T02:04:00.817 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00027509600000000001 2024-04-19T02:04:00.818 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.818 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.818 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:00.818 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.128247+0000", 2024-04-19T02:04:00.818 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.023825774000000001 2024-04-19T02:04:00.818 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.818 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:00.818 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.818 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.818 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.818 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:773 3.ds0 3:b3d82b95:::smithi05930959-4:15 [read 0~3817472,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-04-19T02:04:00.818 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:51.102512+0000", 2024-04-19T02:04:00.818 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.6206018289999999, 2024-04-19T02:04:00.819 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.030365781000000001, 2024-04-19T02:04:00.819 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:00.819 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:00.819 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:04:00.819 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:00.819 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:00.819 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:00.819 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 773 2024-04-19T02:04:00.819 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.819 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:00.819 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.819 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:00.819 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102512+0000", 2024-04-19T02:04:00.819 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.820 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.820 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.820 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:00.820 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102512+0000", 2024-04-19T02:04:00.820 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.820 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.820 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.820 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:00.820 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102513+0000", 2024-04-19T02:04:00.820 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.347e-06 2024-04-19T02:04:00.820 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.820 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.820 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:00.820 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102521+0000", 2024-04-19T02:04:00.820 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5429999999999996e-06 2024-04-19T02:04:00.821 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.821 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.821 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:00.821 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102539+0000", 2024-04-19T02:04:00.821 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7876e-05 2024-04-19T02:04:00.821 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.821 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.821 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.821 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102553+0000", 2024-04-19T02:04:00.821 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4269e-05 2024-04-19T02:04:00.821 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.821 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.821 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:00.821 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.111639+0000", 2024-04-19T02:04:00.821 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0090859500000000006 2024-04-19T02:04:00.822 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.822 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.822 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:00.822 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.111958+0000", 2024-04-19T02:04:00.822 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00031888700000000001 2024-04-19T02:04:00.822 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.822 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.822 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:00.822 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.132878+0000", 2024-04-19T02:04:00.822 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020919909 2024-04-19T02:04:00.822 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.822 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:00.822 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.823 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.823 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.823 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.5s4 67/26 ECSubWrite(tid=86, reqid=client.4509.0:766, at_version=67'42, trim_to=0'0, roll_forward_to=64'39))", 2024-04-19T02:04:00.823 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:51.105381+0000", 2024-04-19T02:04:00.823 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.6177330429999994, 2024-04-19T02:04:00.823 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.027778266999999999, 2024-04-19T02:04:00.823 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:00.823 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:00.823 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:04:00.823 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:00.823 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.823 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:00.823 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.105381+0000", 2024-04-19T02:04:00.823 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.824 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.824 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.824 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:00.824 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.105381+0000", 2024-04-19T02:04:00.824 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:00.824 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.824 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.824 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:00.824 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.105383+0000", 2024-04-19T02:04:00.824 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1900000000000002e-06 2024-04-19T02:04:00.824 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.824 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.824 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:00.824 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.105745+0000", 2024-04-19T02:04:00.824 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00036224299999999999 2024-04-19T02:04:00.825 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.825 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.825 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:00.825 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.105779+0000", 2024-04-19T02:04:00.825 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4310999999999997e-05 2024-04-19T02:04:00.825 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.825 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.825 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:00.825 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.105802+0000", 2024-04-19T02:04:00.825 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2969e-05 2024-04-19T02:04:00.825 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.825 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.825 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:00.825 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.110068+0000", 2024-04-19T02:04:00.825 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0042661180000000002 2024-04-19T02:04:00.826 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.826 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.826 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:00.826 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.110094+0000", 2024-04-19T02:04:00.826 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5814999999999999e-05 2024-04-19T02:04:00.826 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.826 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.826 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:00.826 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.133072+0000", 2024-04-19T02:04:00.826 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022977902000000001 2024-04-19T02:04:00.826 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:00.826 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:00.826 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:00.827 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.133159+0000", 2024-04-19T02:04:00.827 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6718999999999994e-05 2024-04-19T02:04:00.827 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.827 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:00.827 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.827 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:00.827 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:00.827 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:04:00.827 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T02:04:00.829 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-19T02:04:00.830 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:04:00.829+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:04:00.930 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:04:00.929+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:04:00.983 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-04-19T02:04:00.984 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T02:04:01.000 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-04-19T02:04:01.000 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T02:04:01.003 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:04:01.003 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:04:01.003 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-04-19T02:04:01.003 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:04:01.016 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-04-19T02:04:01.017 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T02:04:01.018 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-19T02:04:01.032 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-04-19T02:04:01.033 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T02:04:01.053 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-04-19T02:04:01.053 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T02:04:01.062 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:04:01.065+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:04:01.074 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-04-19T02:04:01.074 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T02:04:01.096 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-04-19T02:04:01.096 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T02:04:01.119 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-04-19T02:04:01.119 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T02:04:01.131 INFO:tasks.rados.rados.0.smithi059.stdout:333: done (13 left) 2024-04-19T02:04:01.131 INFO:tasks.rados.rados.0.smithi059.stdout:334: done (12 left) 2024-04-19T02:04:01.131 INFO:tasks.rados.rados.0.smithi059.stdout:335: done (11 left) 2024-04-19T02:04:01.132 INFO:tasks.rados.rados.0.smithi059.stdout:336: done (10 left) 2024-04-19T02:04:01.132 INFO:tasks.rados.rados.0.smithi059.stdout:337: done (9 left) 2024-04-19T02:04:01.132 INFO:tasks.rados.rados.0.smithi059.stdout:338: done (8 left) 2024-04-19T02:04:01.132 INFO:tasks.rados.rados.0.smithi059.stdout:339: done (7 left) 2024-04-19T02:04:01.132 INFO:tasks.rados.rados.0.smithi059.stdout:341: done (6 left) 2024-04-19T02:04:01.132 INFO:tasks.rados.rados.0.smithi059.stdout:342: done (5 left) 2024-04-19T02:04:01.132 INFO:tasks.rados.rados.0.smithi059.stdout:343: done (4 left) 2024-04-19T02:04:01.132 INFO:tasks.rados.rados.0.smithi059.stdout:344: done (3 left) 2024-04-19T02:04:01.132 INFO:tasks.rados.rados.0.smithi059.stdout:345: done (2 left) 2024-04-19T02:04:01.132 INFO:tasks.rados.rados.0.smithi059.stdout:346: done (1 left) 2024-04-19T02:04:01.132 INFO:tasks.rados.rados.0.smithi059.stdout:347: done (0 left) 2024-04-19T02:04:01.132 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 15 current snap is 29 2024-04-19T02:04:01.132 INFO:tasks.rados.rados.0.smithi059.stdout:348: seq_num 153 ranges {0=163840} 2024-04-19T02:04:01.133 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:04:01.133+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:04:01.134 INFO:tasks.rados.rados.0.smithi059.stdout:348: writing smithi05930959-15 from 0 to 163840 tid 1 2024-04-19T02:04:01.134 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 37 current snap is 29 2024-04-19T02:04:01.134 INFO:tasks.rados.rados.0.smithi059.stdout:349: seq_num 154 ranges {0=180224} 2024-04-19T02:04:01.134 INFO:tasks.rados.rados.0.smithi059.stdout:349: writing smithi05930959-37 from 0 to 180224 tid 1 2024-04-19T02:04:01.134 INFO:tasks.rados.rados.0.smithi059.stdout:350: read oid 47 snap -1 2024-04-19T02:04:01.134 INFO:tasks.rados.rados.0.smithi059.stdout:350: expect (ObjNum 145 snap 24 seq_num 145) 2024-04-19T02:04:01.134 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 6 current snap is 29 2024-04-19T02:04:01.134 INFO:tasks.rados.rados.0.smithi059.stdout:351: seq_num 155 ranges {2408448=212992} 2024-04-19T02:04:01.147 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:04:01.147 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:04:01.147 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-04-19T02:04:01.147 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-04-19T02:04:01.147 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:04:01.151 INFO:tasks.rados.rados.0.smithi059.stdout:351: writing smithi05930959-6 from 2408448 to 2621440 tid 1 2024-04-19T02:04:01.151 INFO:tasks.rados.rados.0.smithi059.stdout:352: rollback oid 45 current snap is 29 2024-04-19T02:04:01.151 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 45 to 21 2024-04-19T02:04:01.151 INFO:tasks.rados.rados.0.smithi059.stdout:350: expect (ObjNum 145 snap 24 seq_num 145) 2024-04-19T02:04:01.152 INFO:tasks.rados.rados.0.smithi059.stdout:348: finishing write tid 1 to smithi05930959-15 2024-04-19T02:04:01.152 INFO:tasks.rados.rados.0.smithi059.stdout:348: finishing write tid 2 to smithi05930959-15 2024-04-19T02:04:01.152 INFO:tasks.rados.rados.0.smithi059.stdout:349: finishing write tid 1 to smithi05930959-37 2024-04-19T02:04:01.152 INFO:tasks.rados.rados.0.smithi059.stdout:349: finishing write tid 2 to smithi05930959-37 2024-04-19T02:04:01.152 INFO:tasks.rados.rados.0.smithi059.stdout:348: finishing write tid 3 to smithi05930959-15 2024-04-19T02:04:01.152 INFO:tasks.rados.rados.0.smithi059.stdout:348: oid 15 updating version 0 to 52 2024-04-19T02:04:01.152 INFO:tasks.rados.rados.0.smithi059.stdout:348: oid 15 updating version 52 to 53 2024-04-19T02:04:01.152 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 15 v 53 (ObjNum 153 snap 29 seq_num 153) dirty exists 2024-04-19T02:04:01.152 INFO:tasks.rados.rados.0.smithi059.stdout:348: left oid 15 (ObjNum 153 snap 29 seq_num 153) 2024-04-19T02:04:01.152 INFO:tasks.rados.rados.0.smithi059.stdout:348: done (4 left) 2024-04-19T02:04:01.153 INFO:tasks.rados.rados.0.smithi059.stdout:350: done (3 left) 2024-04-19T02:04:01.153 INFO:tasks.rados.rados.0.smithi059.stdout:353: copy_from oid 13 from oid 38 current snap is 29 2024-04-19T02:04:01.153 INFO:tasks.rados.rados.0.smithi059.stdout:354: snap_remove snap 20 2024-04-19T02:04:01.153 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-04-19T02:04:01.153 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T02:04:01.167 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-19T02:04:01.177 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-04-19T02:04:01.177 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T02:04:01.197 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-04-19T02:04:01.197 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T02:04:01.217 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-04-19T02:04:01.217 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T02:04:01.232 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:04:01.233+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:04:01.330 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:01.331 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:01.333 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:04:01.333+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:04:01.345 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-19T02:04:01.434 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:04:01.433+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:04:01.467 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:01.468 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:01.483 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-19T02:04:01.535 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:04:01.537+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:04:01.600 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:01.600 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:01.614 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-19T02:04:01.658 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:04:01.661+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:04:01.732 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:01.733 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:01.737 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:04:01.737+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:04:01.747 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-19T02:04:01.838 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:04:01.837+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:04:01.910 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:01.911 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:01.925 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-19T02:04:01.970 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:04:01.973+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:04:02.040 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:04:02.041+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:04:02.043 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:02.043 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:02.053 INFO:tasks.rados.rados.0.smithi059.stdout:354: done (4 left) 2024-04-19T02:04:02.053 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 18 current snap is 29 2024-04-19T02:04:02.053 INFO:tasks.rados.rados.0.smithi059.stdout:349: finishing write tid 3 to smithi05930959-37 2024-04-19T02:04:02.053 INFO:tasks.rados.rados.0.smithi059.stdout:349: oid 37 updating version 0 to 55 2024-04-19T02:04:02.053 INFO:tasks.rados.rados.0.smithi059.stdout:349: oid 37 version 55 is already newer than 54 2024-04-19T02:04:02.053 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 37 v 55 (ObjNum 154 snap 29 seq_num 154) dirty exists 2024-04-19T02:04:02.053 INFO:tasks.rados.rados.0.smithi059.stdout:349: left oid 37 (ObjNum 154 snap 29 seq_num 154) 2024-04-19T02:04:02.053 INFO:tasks.rados.rados.0.smithi059.stdout:355: seq_num 156 ranges {3784704=229376} 2024-04-19T02:04:02.059 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-19T02:04:02.092 INFO:tasks.rados.rados.0.smithi059.stdout:355: writing smithi05930959-18 from 3784704 to 4014080 tid 1 2024-04-19T02:04:02.092 INFO:tasks.rados.rados.0.smithi059.stdout:349: done (4 left) 2024-04-19T02:04:02.092 INFO:tasks.rados.rados.0.smithi059.stdout:356: rmattr oid 49 current snap is 29 2024-04-19T02:04:02.092 INFO:tasks.rados.rados.0.smithi059.stdout:356: done (4 left) 2024-04-19T02:04:02.092 INFO:tasks.rados.rados.0.smithi059.stdout:357: read oid 23 snap -1 2024-04-19T02:04:02.092 INFO:tasks.rados.rados.0.smithi059.stdout:357: expect (ObjNum 94 snap 9 seq_num 94) 2024-04-19T02:04:02.092 INFO:tasks.rados.rados.0.smithi059.stdout:358: copy_from oid 11 from oid 7 current snap is 29 2024-04-19T02:04:02.092 INFO:tasks.rados.rados.0.smithi059.stdout:352: finishing rollback tid 1 to smithi05930959-45 2024-04-19T02:04:02.093 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 45 v 25 (ObjNum 65 snap 0 seq_num 1515086955) dirty dne 2024-04-19T02:04:02.093 INFO:tasks.rados.rados.0.smithi059.stdout:351: finishing write tid 1 to smithi05930959-6 2024-04-19T02:04:02.093 INFO:tasks.rados.rados.0.smithi059.stdout:351: finishing write tid 2 to smithi05930959-6 2024-04-19T02:04:02.093 INFO:tasks.rados.rados.0.smithi059.stdout:351: finishing write tid 3 to smithi05930959-6 2024-04-19T02:04:02.093 INFO:tasks.rados.rados.0.smithi059.stdout:351: oid 6 updating version 0 to 72 2024-04-19T02:04:02.093 INFO:tasks.rados.rados.0.smithi059.stdout:351: oid 6 updating version 72 to 73 2024-04-19T02:04:02.093 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 6 v 73 (ObjNum 155 snap 29 seq_num 155) dirty exists 2024-04-19T02:04:02.093 INFO:tasks.rados.rados.0.smithi059.stdout:351: left oid 6 (ObjNum 155 snap 29 seq_num 155) 2024-04-19T02:04:02.093 INFO:tasks.rados.rados.0.smithi059.stdout:353: finishing copy_from racing read to smithi05930959-13 2024-04-19T02:04:02.093 INFO:tasks.rados.rados.0.smithi059.stdout:351: done (5 left) 2024-04-19T02:04:02.093 INFO:tasks.rados.rados.0.smithi059.stdout:352: done (4 left) 2024-04-19T02:04:02.093 INFO:tasks.rados.rados.0.smithi059.stdout:359: delete oid 17 current snap is 29 2024-04-19T02:04:02.093 INFO:tasks.rados.rados.0.smithi059.stdout:353: finishing copy_from to smithi05930959-13 2024-04-19T02:04:02.093 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 13 v 83 (ObjNum 83 snap 8 seq_num 83) dirty exists 2024-04-19T02:04:02.113 INFO:tasks.rados.rados.0.smithi059.stdout:355: finishing write tid 1 to smithi05930959-18 2024-04-19T02:04:02.114 INFO:tasks.rados.rados.0.smithi059.stdout:355: finishing write tid 2 to smithi05930959-18 2024-04-19T02:04:02.117 INFO:tasks.rados.rados.0.smithi059.stdout:355: finishing write tid 3 to smithi05930959-18 2024-04-19T02:04:02.118 INFO:tasks.rados.rados.0.smithi059.stdout:355: oid 18 updating version 0 to 143 2024-04-19T02:04:02.118 INFO:tasks.rados.rados.0.smithi059.stdout:355: oid 18 version 143 is already newer than 142 2024-04-19T02:04:02.118 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 18 v 143 (ObjNum 156 snap 29 seq_num 156) dirty exists 2024-04-19T02:04:02.118 INFO:tasks.rados.rados.0.smithi059.stdout:355: left oid 18 (ObjNum 156 snap 29 seq_num 156) 2024-04-19T02:04:02.126 INFO:tasks.rados.rados.0.smithi059.stdout:358: finishing copy_from racing read to smithi05930959-11 2024-04-19T02:04:02.127 INFO:tasks.rados.rados.0.smithi059.stdout:357: expect (ObjNum 94 snap 9 seq_num 94) 2024-04-19T02:04:02.140 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:04:02.141+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:04:02.150 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-19T02:04:02.225 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:02.225 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:02.231 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-19T02:04:02.231 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-19T02:04:02.231 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-04-19T02:04:02.231 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T02:04:02.239 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-19T02:04:02.268 INFO:tasks.rados.rados.0.smithi059.stdout:358: finishing copy_from to smithi05930959-11 2024-04-19T02:04:02.268 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 11 v 33 (ObjNum 91 snap 9 seq_num 91) dirty exists 2024-04-19T02:04:02.269 INFO:tasks.rados.rados.0.smithi059.stdout:353: done (4 left) 2024-04-19T02:04:02.269 INFO:tasks.rados.rados.0.smithi059.stdout:355: done (3 left) 2024-04-19T02:04:02.269 INFO:tasks.rados.rados.0.smithi059.stdout:357: done (2 left) 2024-04-19T02:04:02.269 INFO:tasks.rados.rados.0.smithi059.stdout:358: done (1 left) 2024-04-19T02:04:02.269 INFO:tasks.rados.rados.0.smithi059.stdout:359: done (0 left) 2024-04-19T02:04:02.269 INFO:tasks.rados.rados.0.smithi059.stdout:360: snap_remove snap 10 2024-04-19T02:04:02.269 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:04:02.269+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:04:02.342 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:04:02.341+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:04:02.390 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:02.390 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:02.404 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-19T02:04:02.443 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:04:02.441+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:04:02.542 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:04:02.542 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-19T02:04:02.544 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:04:02.545+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:04:02.560 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2024-04-19T02:04:02.221535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":121913888,"num_objects":105,"num_object_clones":56,"num_object_copies":828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":105,"num_whiteouts":11,"num_read":528,"num_read_kb":199910,"num_write":870,"num_write_kb":145290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1044,"ondisk_log_size":1044,"up":146,"acting":146,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1132462080,"kb_used":611184,"kb_used_data":245264,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1131850896,"statfs":{"total":1159641169920,"available":1159015317504,"internally_reserved":0,"allocated":251150336,"data_stored":380920277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7421952,"num_objects":10,"num_object_clones":9,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":-3,"num_read":13,"num_read_kb":9225,"num_write":63,"num_write_kb":5872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000880"},"pg_stats":[{"pgid":"3.f","version":"60'43","reported_seq":290,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-19T02:03:59.125839+0000","last_change":"2024-04-19T02:03:57.073780+0000","last_active":"2024-04-19T02:03:59.125839+0000","last_peered":"2024-04-19T02:03:59.125839+0000","last_clean":"2024-04-19T02:03:59.125839+0000","last_became_active":"2024-04-19T02:03:11.728136+0000","last_became_peered":"2024-04-19T02:03:11.728136+0000","last_unstale":"2024-04-19T02:03:59.125839+0000","last_undegraded":"2024-04-19T02:03:59.125839+0000","last_fullsized":"2024-04-19T02:03:59.125839+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:29.179197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0072430460000000004,"stat_sum":{"num_bytes":8192000,"num_objects":4,"num_object_clones":1,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":15227,"num_write":27,"num_write_kb":9008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,2,3,5,8,0,7],"acting":[11,4,2,3,5,8,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.e","version":"73'54","reported_seq":300,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-19T02:03:59.126047+0000","last_change":"2024-04-19T02:03:57.074482+0000","last_active":"2024-04-19T02:03:59.126047+0000","last_peered":"2024-04-19T02:03:59.126047+0000","last_clean":"2024-04-19T02:03:59.126047+0000","last_became_active":"2024-04-19T02:03:11.730135+0000","last_became_peered":"2024-04-19T02:03:11.730135+0000","last_unstale":"2024-04-19T02:03:59.126047+0000","last_undegraded":"2024-04-19T02:03:59.126047+0000","last_fullsized":"2024-04-19T02:03:59.126047+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:25.211291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017647197,"stat_sum":{"num_bytes":6586368,"num_objects":5,"num_object_clones":2,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":7016,"num_write":27,"num_write_kb":7648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5,11,0,2,3,9],"acting":[7,4,5,11,0,2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.d","version":"74'88","reported_seq":336,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-19T02:03:59.118431+0000","last_change":"2024-04-19T02:03:57.105395+0000","last_active":"2024-04-19T02:03:59.118431+0000","last_peered":"2024-04-19T02:03:59.118431+0000","last_clean":"2024-04-19T02:03:59.118431+0000","last_became_active":"2024-04-19T02:03:11.632682+0000","last_became_peered":"2024-04-19T02:03:11.632682+0000","last_unstale":"2024-04-19T02:03:59.118431+0000","last_undegraded":"2024-04-19T02:03:59.118431+0000","last_fullsized":"2024-04-19T02:03:59.118431+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:15.385331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.049294733,"stat_sum":{"num_bytes":13762560,"num_objects":11,"num_object_clones":7,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":34,"num_read_kb":17085,"num_write":61,"num_write_kb":14496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7,4,5,3,9,0],"acting":[8,11,7,4,5,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.c","version":"73'70","reported_seq":334,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-19T02:03:59.132896+0000","last_change":"2024-04-19T02:03:57.111342+0000","last_active":"2024-04-19T02:03:59.132896+0000","last_peered":"2024-04-19T02:03:59.132896+0000","last_clean":"2024-04-19T02:03:59.132896+0000","last_became_active":"2024-04-19T02:03:11.731568+0000","last_became_peered":"2024-04-19T02:03:11.731568+0000","last_unstale":"2024-04-19T02:03:59.132896+0000","last_undegraded":"2024-04-19T02:03:59.132896+0000","last_fullsized":"2024-04-19T02:03:59.132896+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:32.936893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.030311696999999999,"stat_sum":{"num_bytes":9633792,"num_objects":9,"num_object_clones":5,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":52,"num_read_kb":25410,"num_write":37,"num_write_kb":12960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,11,7,4,2,0,5],"acting":[9,3,11,7,4,2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.b","version":"73'66","reported_seq":313,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-19T02:03:59.116037+0000","last_change":"2024-04-19T02:03:57.093710+0000","last_active":"2024-04-19T02:03:59.116037+0000","last_peered":"2024-04-19T02:03:59.116037+0000","last_clean":"2024-04-19T02:03:59.116037+0000","last_became_active":"2024-04-19T02:03:11.677316+0000","last_became_peered":"2024-04-19T02:03:11.677316+0000","last_unstale":"2024-04-19T02:03:59.116037+0000","last_undegraded":"2024-04-19T02:03:59.116037+0000","last_fullsized":"2024-04-19T02:03:59.116037+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:28.968963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.031744753000000001,"stat_sum":{"num_bytes":4980736,"num_objects":6,"num_object_clones":3,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":33,"num_read_kb":13426,"num_write":32,"num_write_kb":9152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8,6,4,10,1,7],"acting":[3,0,8,6,4,10,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.a","version":"73'84","reported_seq":335,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-19T02:03:59.127178+0000","last_change":"2024-04-19T02:03:57.106475+0000","last_active":"2024-04-19T02:03:59.127178+0000","last_peered":"2024-04-19T02:03:59.127178+0000","last_clean":"2024-04-19T02:03:59.127178+0000","last_became_active":"2024-04-19T02:03:11.727996+0000","last_became_peered":"2024-04-19T02:03:11.727996+0000","last_unstale":"2024-04-19T02:03:59.127178+0000","last_undegraded":"2024-04-19T02:03:59.127178+0000","last_fullsized":"2024-04-19T02:03:59.127178+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:38.671025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.042914552000000002,"stat_sum":{"num_bytes":13926400,"num_objects":9,"num_object_clones":5,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":48922,"num_write":92,"num_write_kb":13600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,1,9,5,7,3,2],"acting":[11,8,1,9,5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.9","version":"64'45","reported_seq":288,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-19T02:03:59.116996+0000","last_change":"2024-04-19T02:03:57.080087+0000","last_active":"2024-04-19T02:03:59.116996+0000","last_peered":"2024-04-19T02:03:59.116996+0000","last_clean":"2024-04-19T02:03:59.116996+0000","last_became_active":"2024-04-19T02:03:11.729255+0000","last_became_peered":"2024-04-19T02:03:11.729255+0000","last_unstale":"2024-04-19T02:03:59.116996+0000","last_undegraded":"2024-04-19T02:03:59.116996+0000","last_fullsized":"2024-04-19T02:03:59.116996+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:35.637052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007028337,"stat_sum":{"num_bytes":6012928,"num_objects":6,"num_object_clones":3,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":21,"num_read_kb":9226,"num_write":30,"num_write_kb":8576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,10,2,9,1,7,11],"acting":[4,3,10,2,9,1,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.8","version":"73'77","reported_seq":328,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-19T02:03:59.116648+0000","last_change":"2024-04-19T02:03:57.111667+0000","last_active":"2024-04-19T02:03:59.116648+0000","last_peered":"2024-04-19T02:03:59.116648+0000","last_clean":"2024-04-19T02:03:59.116648+0000","last_became_active":"2024-04-19T02:03:11.679384+0000","last_became_peered":"2024-04-19T02:03:11.679384+0000","last_unstale":"2024-04-19T02:03:59.116648+0000","last_undegraded":"2024-04-19T02:03:59.116648+0000","last_fullsized":"2024-04-19T02:03:59.116648+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:15.108203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.040712247999999999,"stat_sum":{"num_bytes":12058624,"num_objects":8,"num_object_clones":5,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":35728,"num_write":39,"num_write_kb":11920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7,11,5,9,6,10],"acting":[4,1,7,11,5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"1.0","version":"21'137","reported_seq":295,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-19T02:03:59.117599+0000","last_change":"2024-04-19T02:02:59.974166+0000","last_active":"2024-04-19T02:03:59.117599+0000","last_peered":"2024-04-19T02:03:59.117599+0000","last_clean":"2024-04-19T02:03:59.117599+0000","last_became_active":"2024-04-19T02:02:59.972005+0000","last_became_peered":"2024-04-19T02:02:59.972005+0000","last_unstale":"2024-04-19T02:03:59.117599+0000","last_undegraded":"2024-04-19T02:03:59.117599+0000","last_fullsized":"2024-04-19T02:03:59.117599+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:02:58.486225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:02:58.486225+0000","last_clean_scrub_stamp":"2024-04-19T02:02:58.486225+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:20.275707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":115,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-19T02:03:59.125518+0000","last_change":"2024-04-19T02:03:03.026806+0000","last_active":"2024-04-19T02:03:59.125518+0000","last_peered":"2024-04-19T02:03:59.125518+0000","last_clean":"2024-04-19T02:03:59.125518+0000","last_became_active":"2024-04-19T02:03:03.026063+0000","last_became_peered":"2024-04-19T02:03:03.026063+0000","last_unstale":"2024-04-19T02:03:59.125518+0000","last_undegraded":"2024-04-19T02:03:59.125518+0000","last_fullsized":"2024-04-19T02:03:59.125518+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:19.673016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"73'147","reported_seq":399,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-19T02:03:59.115955+0000","last_change":"2024-04-19T02:03:57.094921+0000","last_active":"2024-04-19T02:03:59.115955+0000","last_peered":"2024-04-19T02:03:59.115955+0000","last_clean":"2024-04-19T02:03:59.115955+0000","last_became_active":"2024-04-19T02:03:11.630475+0000","last_became_peered":"2024-04-19T02:03:11.630475+0000","last_unstale":"2024-04-19T02:03:59.115955+0000","last_undegraded":"2024-04-19T02:03:59.115955+0000","last_fullsized":"2024-04-19T02:03:59.115955+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:25.596487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.037385165999999997,"stat_sum":{"num_bytes":18120704,"num_objects":14,"num_object_clones":7,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":2,"num_read":55,"num_read_kb":14365,"num_write":108,"num_write_kb":23520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10,4,8,1,11,9],"acting":[3,5,10,4,8,1,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":116,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-19T02:03:59.117575+0000","last_change":"2024-04-19T02:03:02.561076+0000","last_active":"2024-04-19T02:03:59.117575+0000","last_peered":"2024-04-19T02:03:59.117575+0000","last_clean":"2024-04-19T02:03:59.117575+0000","last_became_active":"2024-04-19T02:03:02.560210+0000","last_became_peered":"2024-04-19T02:03:02.560210+0000","last_unstale":"2024-04-19T02:03:59.117575+0000","last_undegraded":"2024-04-19T02:03:59.117575+0000","last_fullsized":"2024-04-19T02:03:59.117575+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:16.900004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"76'35","reported_seq":286,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-19T02:04:00.128957+0000","last_change":"2024-04-19T02:03:57.093654+0000","last_active":"2024-04-19T02:04:00.128957+0000","last_peered":"2024-04-19T02:04:00.128957+0000","last_clean":"2024-04-19T02:04:00.128957+0000","last_became_active":"2024-04-19T02:03:11.633394+0000","last_became_peered":"2024-04-19T02:03:11.633394+0000","last_unstale":"2024-04-19T02:04:00.128957+0000","last_undegraded":"2024-04-19T02:04:00.128957+0000","last_fullsized":"2024-04-19T02:04:00.128957+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:11.422843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033663516999999997,"stat_sum":{"num_bytes":6537216,"num_objects":5,"num_object_clones":3,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":23,"num_read_kb":4042,"num_write":43,"num_write_kb":6912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,4,5,10,7,9,1],"acting":[0,11,4,5,10,7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":111,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-19T02:03:59.115699+0000","last_change":"2024-04-19T02:03:09.594947+0000","last_active":"2024-04-19T02:03:59.115699+0000","last_peered":"2024-04-19T02:03:59.115699+0000","last_clean":"2024-04-19T02:03:59.115699+0000","last_became_active":"2024-04-19T02:03:09.594072+0000","last_became_peered":"2024-04-19T02:03:09.594072+0000","last_unstale":"2024-04-19T02:03:59.115699+0000","last_undegraded":"2024-04-19T02:03:59.115699+0000","last_fullsized":"2024-04-19T02:03:59.115699+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:29.461650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":232,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-19T02:03:58.151791+0000","last_change":"2024-04-19T02:03:57.086382+0000","last_active":"2024-04-19T02:03:58.151791+0000","last_peered":"2024-04-19T02:03:58.151791+0000","last_clean":"2024-04-19T02:03:58.151791+0000","last_became_active":"2024-04-19T02:03:11.678078+0000","last_became_peered":"2024-04-19T02:03:11.678078+0000","last_unstale":"2024-04-19T02:03:58.151791+0000","last_undegraded":"2024-04-19T02:03:58.151791+0000","last_fullsized":"2024-04-19T02:03:58.151791+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:27.404569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018400757,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,9,1,6,4,0,10],"acting":[5,7,9,1,6,4,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":116,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-19T02:03:59.117914+0000","last_change":"2024-04-19T02:03:02.560133+0000","last_active":"2024-04-19T02:03:59.117914+0000","last_peered":"2024-04-19T02:03:59.117914+0000","last_clean":"2024-04-19T02:03:59.117914+0000","last_became_active":"2024-04-19T02:03:02.559377+0000","last_became_peered":"2024-04-19T02:03:02.559377+0000","last_unstale":"2024-04-19T02:03:59.117914+0000","last_undegraded":"2024-04-19T02:03:59.117914+0000","last_fullsized":"2024-04-19T02:03:59.117914+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:30.728400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"74'27","reported_seq":262,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-19T02:03:58.219846+0000","last_change":"2024-04-19T02:03:57.072989+0000","last_active":"2024-04-19T02:03:58.219846+0000","last_peered":"2024-04-19T02:03:58.219846+0000","last_clean":"2024-04-19T02:03:58.219846+0000","last_became_active":"2024-04-19T02:03:11.677099+0000","last_became_peered":"2024-04-19T02:03:11.677099+0000","last_unstale":"2024-04-19T02:03:58.219846+0000","last_undegraded":"2024-04-19T02:03:58.219846+0000","last_fullsized":"2024-04-19T02:03:58.219846+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:29.372259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087926659999999993,"stat_sum":{"num_bytes":4390912,"num_objects":4,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":2370,"num_write":55,"num_write_kb":4288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,0,3,4,7,9,6],"acting":[10,8,0,3,4,7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"3.5","version":"73'47","reported_seq":278,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-19T02:03:58.161868+0000","last_change":"2024-04-19T02:03:57.105632+0000","last_active":"2024-04-19T02:03:58.161868+0000","last_peered":"2024-04-19T02:03:58.161868+0000","last_clean":"2024-04-19T02:03:58.161868+0000","last_became_active":"2024-04-19T02:03:11.728949+0000","last_became_peered":"2024-04-19T02:03:11.728949+0000","last_unstale":"2024-04-19T02:03:58.161868+0000","last_undegraded":"2024-04-19T02:03:58.161868+0000","last_fullsized":"2024-04-19T02:03:58.161868+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:32.976707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.042418823000000001,"stat_sum":{"num_bytes":4620288,"num_objects":4,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":9,"num_read_kb":6,"num_write":42,"num_write_kb":5856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2,9,8,1,6,5],"acting":[10,3,2,9,8,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":114,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-19T02:03:58.156016+0000","last_change":"2024-04-19T02:03:02.597582+0000","last_active":"2024-04-19T02:03:58.156016+0000","last_peered":"2024-04-19T02:03:58.156016+0000","last_clean":"2024-04-19T02:03:58.156016+0000","last_became_active":"2024-04-19T02:03:02.597118+0000","last_became_peered":"2024-04-19T02:03:02.597118+0000","last_unstale":"2024-04-19T02:03:58.156016+0000","last_undegraded":"2024-04-19T02:03:58.156016+0000","last_fullsized":"2024-04-19T02:03:58.156016+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:13.683143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"74'91","reported_seq":328,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-19T02:03:58.182892+0000","last_change":"2024-04-19T02:03:57.109118+0000","last_active":"2024-04-19T02:03:58.182892+0000","last_peered":"2024-04-19T02:03:58.182892+0000","last_clean":"2024-04-19T02:03:58.182892+0000","last_became_active":"2024-04-19T02:03:11.730165+0000","last_became_peered":"2024-04-19T02:03:11.730165+0000","last_unstale":"2024-04-19T02:03:58.182892+0000","last_undegraded":"2024-04-19T02:03:58.182892+0000","last_fullsized":"2024-04-19T02:03:58.182892+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:29.723395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.055816989999999997,"stat_sum":{"num_bytes":10158080,"num_objects":13,"num_object_clones":8,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":29,"num_read_kb":6446,"num_write":64,"num_write_kb":11744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,10,8,7,2,11,0],"acting":[1,5,10,8,7,2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":116,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-19T02:03:59.125788+0000","last_change":"2024-04-19T02:03:02.595929+0000","last_active":"2024-04-19T02:03:59.125788+0000","last_peered":"2024-04-19T02:03:59.125788+0000","last_clean":"2024-04-19T02:03:59.125788+0000","last_became_active":"2024-04-19T02:03:02.595232+0000","last_became_peered":"2024-04-19T02:03:02.595232+0000","last_unstale":"2024-04-19T02:03:59.125788+0000","last_undegraded":"2024-04-19T02:03:59.125788+0000","last_fullsized":"2024-04-19T02:03:59.125788+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:04.077370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"64'26","reported_seq":260,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-19T02:03:59.116404+0000","last_change":"2024-04-19T02:03:57.059686+0000","last_active":"2024-04-19T02:03:59.116404+0000","last_peered":"2024-04-19T02:03:59.116404+0000","last_clean":"2024-04-19T02:03:59.116404+0000","last_became_active":"2024-04-19T02:03:11.675056+0000","last_became_peered":"2024-04-19T02:03:11.675056+0000","last_unstale":"2024-04-19T02:03:59.116404+0000","last_undegraded":"2024-04-19T02:03:59.116404+0000","last_fullsized":"2024-04-19T02:03:59.116404+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:18.696455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001771391,"stat_sum":{"num_bytes":2162688,"num_objects":3,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":5,"num_read_kb":2,"num_write":9,"num_write_kb":2496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,9,0,7,5,11,1],"acting":[3,6,9,0,7,5,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":114,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-19T02:03:58.155434+0000","last_change":"2024-04-19T02:03:02.988302+0000","last_active":"2024-04-19T02:03:58.155434+0000","last_peered":"2024-04-19T02:03:58.155434+0000","last_clean":"2024-04-19T02:03:58.155434+0000","last_became_active":"2024-04-19T02:03:02.988073+0000","last_became_peered":"2024-04-19T02:03:02.988073+0000","last_unstale":"2024-04-19T02:03:58.155434+0000","last_undegraded":"2024-04-19T02:03:58.155434+0000","last_fullsized":"2024-04-19T02:03:58.155434+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:28.455271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"74'7","reported_seq":248,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-19T02:04:00.128508+0000","last_change":"2024-04-19T02:03:57.064435+0000","last_active":"2024-04-19T02:04:00.128508+0000","last_peered":"2024-04-19T02:04:00.128508+0000","last_clean":"2024-04-19T02:04:00.128508+0000","last_became_active":"2024-04-19T02:03:11.673940+0000","last_became_peered":"2024-04-19T02:03:11.673940+0000","last_unstale":"2024-04-19T02:04:00.128508+0000","last_undegraded":"2024-04-19T02:04:00.128508+0000","last_fullsized":"2024-04-19T02:04:00.128508+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:39.967245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029895170000000001,"stat_sum":{"num_bytes":180224,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":530,"num_write":23,"num_write_kb":176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9,8,4,5,6,10],"acting":[0,1,9,8,4,5,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":116,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-19T02:03:59.118181+0000","last_change":"2024-04-19T02:03:02.570839+0000","last_active":"2024-04-19T02:03:59.118181+0000","last_peered":"2024-04-19T02:03:59.118181+0000","last_clean":"2024-04-19T02:03:59.118181+0000","last_became_active":"2024-04-19T02:03:02.570116+0000","last_became_peered":"2024-04-19T02:03:02.570116+0000","last_unstale":"2024-04-19T02:03:59.118181+0000","last_undegraded":"2024-04-19T02:03:59.118181+0000","last_fullsized":"2024-04-19T02:03:59.118181+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:23.134425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":121323520,"num_objects":103,"num_object_clones":56,"num_object_copies":824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":11,"num_read":402,"num_read_kb":199801,"num_write":689,"num_write_kb":142352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":240877568,"data_stored":380235776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":907,"ondisk_log_size":907,"up":128,"acting":128,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":19,"seq":81604378640,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46456,"kb_used_data":15276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325384,"statfs":{"total":96636764160,"available":96589193216,"internally_reserved":0,"allocated":15642624,"data_stored":24571684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":19,"seq":81604378639,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48016,"kb_used_data":16832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323824,"statfs":{"total":96636764160,"available":96587595776,"internally_reserved":0,"allocated":17235968,"data_stored":30314501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411343,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57824,"kb_used_data":26644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94314016,"statfs":{"total":96636764160,"available":96577552384,"internally_reserved":0,"allocated":27283456,"data_stored":40529475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411344,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55604,"kb_used_data":24420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94316236,"statfs":{"total":96636764160,"available":96579825664,"internally_reserved":0,"allocated":25006080,"data_stored":38973220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411344,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51316,"kb_used_data":20128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320524,"statfs":{"total":96636764160,"available":96584216576,"internally_reserved":0,"allocated":20611072,"data_stored":31858787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411344,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55696,"kb_used_data":24508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94316144,"statfs":{"total":96636764160,"available":96579731456,"internally_reserved":0,"allocated":25096192,"data_stored":39579203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":18,"seq":77309411342,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34792,"kb_used_data":7760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337048,"statfs":{"total":96636764160,"available":96601137152,"internally_reserved":0,"allocated":7946240,"data_stored":11845187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411343,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53760,"kb_used_data":22568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94318080,"statfs":{"total":96636764160,"available":96581713920,"internally_reserved":0,"allocated":23109632,"data_stored":32230979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":18,"seq":77309411343,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54228,"kb_used_data":23040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94317612,"statfs":{"total":96636764160,"available":96581234688,"internally_reserved":0,"allocated":23592960,"data_stored":35208771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":17,"seq":73014444047,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44232,"kb_used_data":17200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94327608,"statfs":{"total":96636764160,"available":96591470592,"internally_reserved":0,"allocated":17612800,"data_stored":24989251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":17,"seq":73014444047,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58052,"kb_used_data":26868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94313788,"statfs":{"total":96636764160,"available":96577318912,"internally_reserved":0,"allocated":27512832,"data_stored":40886371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":17,"seq":73014444047,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51208,"kb_used_data":20020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320632,"statfs":{"total":96636764160,"available":96584327168,"internally_reserved":0,"allocated":20500480,"data_stored":29932848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16363520,"data_stored":29782016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19689472,"data_stored":30040064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":15761408,"data_stored":26984448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":22745088,"data_stored":35696640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":22261760,"data_stored":31715328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":26132480,"data_stored":39841792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":7098368,"data_stored":12333056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24248320,"data_stored":39071744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":19230720,"data_stored":31825920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":24145920,"data_stored":38449152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":16764928,"data_stored":24473600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":26435584,"data_stored":40022016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T02:04:02.562 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pgp_num 16 2024-04-19T02:04:02.571 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:02.572 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:02.586 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-19T02:04:02.671 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:04:02.673+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:04:02.745 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:02.745 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:02.745 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:04:02.745+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:04:02.759 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-19T02:04:02.846 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:04:02.845+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:04:02.938 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:02.938 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:02.947 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:04:02.949+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:04:02.952 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-19T02:04:03.048 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:04:03.049+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:04:03.076 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:03.076 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:03.090 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-19T02:04:03.137 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is unset 2024-04-19T02:04:03.137 INFO:teuthology.orchestra.run.smithi059.stderr:set pool 3 pgp_num to 16 2024-04-19T02:04:03.146 INFO:tasks.rados.rados.0.smithi059.stdout:360: done (0 left) 2024-04-19T02:04:03.146 INFO:tasks.rados.rados.0.smithi059.stdout:361: copy_from oid 47 from oid 23 current snap is 29 2024-04-19T02:04:03.147 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 21 current snap is 29 2024-04-19T02:04:03.147 INFO:tasks.rados.rados.0.smithi059.stdout:362: seq_num 158 ranges {4079616=262144} 2024-04-19T02:04:03.156 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:04:03.149+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:04:03.173 INFO:tasks.rados.rados.0.smithi059.stdout:362: writing smithi05930959-21 from 4079616 to 4341760 tid 1 2024-04-19T02:04:03.173 INFO:tasks.rados.rados.0.smithi059.stdout:363: snap_remove snap 16 2024-04-19T02:04:03.237 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T02:04:03.250 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:04:03.249+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:04:03.287 INFO:teuthology.orchestra.run.smithi059.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:03.288 INFO:teuthology.orchestra.run.smithi059.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:03.303 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-19T02:04:03.351 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:04:03.353+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:04:03.377 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-04-19T02:04:03.377 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T02:04:03.396 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-04-19T02:04:03.396 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T02:04:03.412 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-04-19T02:04:03.412 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T02:04:03.429 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:04:03.429 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:04:03.430 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-04-19T02:04:03.430 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:04:03.430 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-04-19T02:04:03.430 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T02:04:03.444 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-19T02:04:03.452 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-04-19T02:04:03.452 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T02:04:03.452 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:04:03.453+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:04:03.472 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-04-19T02:04:03.473 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T02:04:03.495 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-04-19T02:04:03.495 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T02:04:03.517 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-04-19T02:04:03.517 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T02:04:03.540 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-04-19T02:04:03.540 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T02:04:03.553 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:04:03.553+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:04:03.562 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:04:03.563 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:04:03.563 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-04-19T02:04:03.563 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-04-19T02:04:03.563 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:04:03.563 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-04-19T02:04:03.563 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T02:04:03.577 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-19T02:04:03.586 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-04-19T02:04:03.587 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T02:04:03.611 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-04-19T02:04:03.611 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T02:04:03.654 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:04:03.657+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:04:03.708 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:04:03.708 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-04-19T02:04:03.708 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-04-19T02:04:03.708 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-04-19T02:04:03.708 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.708 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 40/26 ECSubWrite(tid=30, reqid=client.4509.0:359, at_version=40'16, trim_to=0'0, roll_forward_to=39'14))", 2024-04-19T02:04:03.708 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.790646+0000", 2024-04-19T02:04:03.709 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 38.917533362, 2024-04-19T02:04:03.709 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.004294422, 2024-04-19T02:04:03.709 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:03.709 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:03.709 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:03.709 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:03.709 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.709 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:03.709 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.790646+0000", 2024-04-19T02:04:03.709 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.709 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.709 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.709 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:03.709 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.790646+0000", 2024-04-19T02:04:03.710 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.710 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.710 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.710 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:03.710 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.790647+0000", 2024-04-19T02:04:03.710 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7400000000000003e-07 2024-04-19T02:04:03.710 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.710 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.710 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:03.710 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.791098+0000", 2024-04-19T02:04:03.710 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00045124599999999998 2024-04-19T02:04:03.710 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.710 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.710 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:03.711 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.791116+0000", 2024-04-19T02:04:03.711 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7351999999999999e-05 2024-04-19T02:04:03.711 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.711 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.711 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:03.711 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.791126+0000", 2024-04-19T02:04:03.711 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0606e-05 2024-04-19T02:04:03.711 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.711 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.711 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:03.711 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.791206+0000", 2024-04-19T02:04:03.711 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0018999999999994e-05 2024-04-19T02:04:03.711 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.711 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.711 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:03.712 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.791217+0000", 2024-04-19T02:04:03.712 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0361e-05 2024-04-19T02:04:03.712 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.712 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.712 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:03.712 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.794897+0000", 2024-04-19T02:04:03.712 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0036802810000000001 2024-04-19T02:04:03.712 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.712 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.712 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:03.712 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.794941+0000", 2024-04-19T02:04:03.712 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.3983000000000002e-05 2024-04-19T02:04:03.712 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.712 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:03.712 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.713 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.713 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.713 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s5 42/26 ECSubWrite(tid=43, reqid=osd.1.0:43, at_version=42'30, trim_to=0'0, roll_forward_to=40'28))", 2024-04-19T02:04:03.713 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:26.737676+0000", 2024-04-19T02:04:03.713 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 36.970503698000002, 2024-04-19T02:04:03.713 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0079882579999999998, 2024-04-19T02:04:03.713 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:03.713 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:03.713 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:03.713 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:03.713 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.713 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:03.713 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.737676+0000", 2024-04-19T02:04:03.713 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.714 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.714 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.714 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:03.714 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.737676+0000", 2024-04-19T02:04:03.714 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.714 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.714 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.714 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:03.714 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.737676+0000", 2024-04-19T02:04:03.714 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3999999999999997e-07 2024-04-19T02:04:03.714 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.714 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.714 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:03.714 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.737712+0000", 2024-04-19T02:04:03.715 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5405999999999999e-05 2024-04-19T02:04:03.715 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.715 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.715 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:03.715 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.737731+0000", 2024-04-19T02:04:03.715 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9437e-05 2024-04-19T02:04:03.715 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.715 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.715 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:03.715 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.737746+0000", 2024-04-19T02:04:03.715 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5013e-05 2024-04-19T02:04:03.715 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.715 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.715 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:03.716 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.737887+0000", 2024-04-19T02:04:03.716 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014084300000000001 2024-04-19T02:04:03.716 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.716 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.716 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:03.716 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.737902+0000", 2024-04-19T02:04:03.716 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5138e-05 2024-04-19T02:04:03.716 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.716 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.716 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:03.716 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.745612+0000", 2024-04-19T02:04:03.716 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0077100049999999998 2024-04-19T02:04:03.716 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.716 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.716 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:03.717 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.745664+0000", 2024-04-19T02:04:03.717 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.2076000000000001e-05 2024-04-19T02:04:03.717 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.717 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:03.717 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.717 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.717 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.717 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s5 42/26 ECSubWrite(tid=44, reqid=osd.1.0:44, at_version=42'32, trim_to=0'0, roll_forward_to=40'28))", 2024-04-19T02:04:03.717 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:26.739008+0000", 2024-04-19T02:04:03.717 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 36.969172090999997, 2024-04-19T02:04:03.717 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0066868379999999996, 2024-04-19T02:04:03.717 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:03.717 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:03.717 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:03.717 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:03.718 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.718 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:03.718 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.739008+0000", 2024-04-19T02:04:03.718 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.718 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.718 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.718 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:03.718 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.739008+0000", 2024-04-19T02:04:03.718 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.718 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.718 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.718 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:03.718 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.739009+0000", 2024-04-19T02:04:03.718 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.4300000000000001e-07 2024-04-19T02:04:03.718 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.719 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.719 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:03.719 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.739041+0000", 2024-04-19T02:04:03.719 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2642999999999999e-05 2024-04-19T02:04:03.719 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.719 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.719 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:03.719 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.739057+0000", 2024-04-19T02:04:03.719 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6124e-05 2024-04-19T02:04:03.719 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.719 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.719 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:03.719 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.739067+0000", 2024-04-19T02:04:03.719 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3389999999999999e-06 2024-04-19T02:04:03.720 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.720 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.720 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:03.720 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.739141+0000", 2024-04-19T02:04:03.720 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.3794999999999994e-05 2024-04-19T02:04:03.720 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.720 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.720 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:03.720 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.739152+0000", 2024-04-19T02:04:03.720 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0954e-05 2024-04-19T02:04:03.720 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.720 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.720 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:03.720 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.745671+0000", 2024-04-19T02:04:03.720 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0065197450000000004 2024-04-19T02:04:03.721 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.721 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.721 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:03.721 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.745695+0000", 2024-04-19T02:04:03.721 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3295e-05 2024-04-19T02:04:03.721 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.721 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:03.721 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.721 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.721 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.721 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.5s2 42/26 ECSubWrite(tid=51, reqid=osd.10.0:51, at_version=42'27, trim_to=0'0, roll_forward_to=39'25))", 2024-04-19T02:04:03.721 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:26.748457+0000", 2024-04-19T02:04:03.721 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 36.959722722999999, 2024-04-19T02:04:03.721 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0043907169999999997, 2024-04-19T02:04:03.722 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:03.722 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:03.722 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:03.722 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:03.722 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.722 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:03.722 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.748457+0000", 2024-04-19T02:04:03.722 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.722 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.722 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.722 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:03.723 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.748457+0000", 2024-04-19T02:04:03.723 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.723 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.723 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.723 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:03.723 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.748458+0000", 2024-04-19T02:04:03.723 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0699999999999998e-07 2024-04-19T02:04:03.723 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.723 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.723 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:03.723 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.748491+0000", 2024-04-19T02:04:03.723 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.392e-05 2024-04-19T02:04:03.723 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.723 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.724 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:03.724 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.748509+0000", 2024-04-19T02:04:03.724 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7450000000000001e-05 2024-04-19T02:04:03.724 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.724 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.724 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:03.724 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.748521+0000", 2024-04-19T02:04:03.724 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2591e-05 2024-04-19T02:04:03.724 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.724 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.724 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:03.724 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.748598+0000", 2024-04-19T02:04:03.724 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6886999999999998e-05 2024-04-19T02:04:03.724 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.724 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.725 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:03.725 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.748608+0000", 2024-04-19T02:04:03.725 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.6709999999999994e-06 2024-04-19T02:04:03.725 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.725 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.725 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:03.725 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.752792+0000", 2024-04-19T02:04:03.725 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0041843339999999996 2024-04-19T02:04:03.725 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.725 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.725 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:03.725 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.752848+0000", 2024-04-19T02:04:03.725 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5457000000000001e-05 2024-04-19T02:04:03.725 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.726 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:03.726 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.726 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.726 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.726 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.as7 42/26 ECSubWrite(tid=82, reqid=osd.11.0:82, at_version=42'34, trim_to=0'0, roll_forward_to=39'32))", 2024-04-19T02:04:03.726 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:26.757483+0000", 2024-04-19T02:04:03.726 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 36.950697089000002, 2024-04-19T02:04:03.726 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0054335750000000004, 2024-04-19T02:04:03.726 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:03.726 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:03.726 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:03.726 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:03.726 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.726 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:03.726 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757483+0000", 2024-04-19T02:04:03.727 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.727 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.727 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.727 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:03.727 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757483+0000", 2024-04-19T02:04:03.727 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.727 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.727 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.727 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:03.727 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757483+0000", 2024-04-19T02:04:03.727 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1e-07 2024-04-19T02:04:03.727 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.727 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.727 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:03.728 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757508+0000", 2024-04-19T02:04:03.728 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4681999999999998e-05 2024-04-19T02:04:03.728 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.728 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.728 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:03.728 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757525+0000", 2024-04-19T02:04:03.728 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6869000000000001e-05 2024-04-19T02:04:03.728 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.728 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.728 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:03.728 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757535+0000", 2024-04-19T02:04:03.728 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0074999999999999e-05 2024-04-19T02:04:03.728 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.728 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.728 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:03.729 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757714+0000", 2024-04-19T02:04:03.729 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000179388 2024-04-19T02:04:03.729 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.729 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.729 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:03.729 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757729+0000", 2024-04-19T02:04:03.729 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4703e-05 2024-04-19T02:04:03.729 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.729 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.729 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:03.729 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.762889+0000", 2024-04-19T02:04:03.729 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0051599430000000002 2024-04-19T02:04:03.729 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.729 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.730 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:03.730 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.762916+0000", 2024-04-19T02:04:03.730 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7605000000000001e-05 2024-04-19T02:04:03.730 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.730 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:03.730 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.730 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.730 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.730 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs2 42/26 ECSubWrite(tid=83, reqid=osd.11.0:83, at_version=42'26, trim_to=0'0, roll_forward_to=39'24))", 2024-04-19T02:04:03.730 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:26.757561+0000", 2024-04-19T02:04:03.730 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 36.950619211000003, 2024-04-19T02:04:03.730 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0053118139999999998, 2024-04-19T02:04:03.730 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:03.731 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:03.731 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:03.731 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:03.731 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.731 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:03.731 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757561+0000", 2024-04-19T02:04:03.731 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.731 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.731 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.731 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:03.731 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757561+0000", 2024-04-19T02:04:03.731 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.731 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.731 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.731 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:03.732 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757561+0000", 2024-04-19T02:04:03.732 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.58e-07 2024-04-19T02:04:03.732 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.732 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.732 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:03.732 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757602+0000", 2024-04-19T02:04:03.732 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0790000000000001e-05 2024-04-19T02:04:03.732 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.732 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.732 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:03.732 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757612+0000", 2024-04-19T02:04:03.732 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0078000000000001e-05 2024-04-19T02:04:03.732 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.732 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.732 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:03.733 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757648+0000", 2024-04-19T02:04:03.733 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6046e-05 2024-04-19T02:04:03.733 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.733 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.733 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:03.733 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757718+0000", 2024-04-19T02:04:03.733 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9772999999999998e-05 2024-04-19T02:04:03.733 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.733 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.733 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:03.733 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757727+0000", 2024-04-19T02:04:03.733 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.7280000000000007e-06 2024-04-19T02:04:03.733 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.733 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.734 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:03.734 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.762829+0000", 2024-04-19T02:04:03.734 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0051016539999999997 2024-04-19T02:04:03.734 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.734 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.734 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:03.734 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.762872+0000", 2024-04-19T02:04:03.734 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.3387000000000003e-05 2024-04-19T02:04:03.734 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.734 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:03.734 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.734 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.734 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.734 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.5s2 42/26 ECSubWrite(tid=52, reqid=osd.10.0:52, at_version=42'29, trim_to=0'0, roll_forward_to=39'25))", 2024-04-19T02:04:03.735 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:26.759271+0000", 2024-04-19T02:04:03.735 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 36.948909188000002, 2024-04-19T02:04:03.735 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0041464099999999997, 2024-04-19T02:04:03.735 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:03.735 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:03.735 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:03.735 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:03.735 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.735 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:03.735 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.759271+0000", 2024-04-19T02:04:03.735 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.735 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.735 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.735 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:03.736 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.759271+0000", 2024-04-19T02:04:03.736 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.736 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.736 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.736 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:03.736 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.759272+0000", 2024-04-19T02:04:03.736 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2240000000000001e-06 2024-04-19T02:04:03.736 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.736 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.736 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:03.736 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.759296+0000", 2024-04-19T02:04:03.736 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4555000000000001e-05 2024-04-19T02:04:03.736 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.736 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.736 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:03.737 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.759317+0000", 2024-04-19T02:04:03.737 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0384999999999999e-05 2024-04-19T02:04:03.737 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.737 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.737 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:03.737 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.759326+0000", 2024-04-19T02:04:03.737 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7819999999999999e-06 2024-04-19T02:04:03.737 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.737 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.737 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:03.737 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.759406+0000", 2024-04-19T02:04:03.737 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0458999999999997e-05 2024-04-19T02:04:03.737 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.737 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.738 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:03.738 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.759419+0000", 2024-04-19T02:04:03.738 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3050999999999999e-05 2024-04-19T02:04:03.738 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.738 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.738 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:03.738 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.763377+0000", 2024-04-19T02:04:03.738 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0039574299999999996 2024-04-19T02:04:03.738 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.738 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.738 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:03.738 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.763417+0000", 2024-04-19T02:04:03.738 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0524e-05 2024-04-19T02:04:03.738 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.739 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:03.739 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.739 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.739 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.739 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 42/26 ECSubWrite(tid=32, reqid=osd.9.0:32, at_version=42'18, trim_to=0'0, roll_forward_to=40'16))", 2024-04-19T02:04:03.739 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:26.765866+0000", 2024-04-19T02:04:03.739 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 36.942313595000002, 2024-04-19T02:04:03.739 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0081652840000000001, 2024-04-19T02:04:03.739 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:03.739 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:03.739 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:03.739 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:03.739 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.739 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:03.740 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.765866+0000", 2024-04-19T02:04:03.740 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.740 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.740 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.740 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:03.740 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.765866+0000", 2024-04-19T02:04:03.740 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.740 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.740 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.740 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:03.740 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.765867+0000", 2024-04-19T02:04:03.740 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7599999999999998e-07 2024-04-19T02:04:03.740 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.740 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.740 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:03.741 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.765890+0000", 2024-04-19T02:04:03.741 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3459e-05 2024-04-19T02:04:03.741 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.741 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.741 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:03.741 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.765901+0000", 2024-04-19T02:04:03.741 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1426e-05 2024-04-19T02:04:03.741 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.741 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.741 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:03.741 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.765911+0000", 2024-04-19T02:04:03.741 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.2969999999999992e-06 2024-04-19T02:04:03.741 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.741 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.741 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:03.742 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.765996+0000", 2024-04-19T02:04:03.742 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5568999999999993e-05 2024-04-19T02:04:03.742 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.742 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.742 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:03.742 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.766011+0000", 2024-04-19T02:04:03.742 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4615e-05 2024-04-19T02:04:03.742 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.742 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.742 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:03.742 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.773989+0000", 2024-04-19T02:04:03.742 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0079778669999999996 2024-04-19T02:04:03.742 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.742 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.743 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:03.743 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.774032+0000", 2024-04-19T02:04:03.743 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2775000000000003e-05 2024-04-19T02:04:03.743 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.743 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:03.743 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.743 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.743 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.743 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 46/26 ECSubWrite(tid=44, reqid=osd.9.0:44, at_version=46'26, trim_to=0'0, roll_forward_to=45'24))", 2024-04-19T02:04:03.743 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.065734+0000", 2024-04-19T02:04:03.743 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 33.642445827000003, 2024-04-19T02:04:03.743 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0090304430000000008, 2024-04-19T02:04:03.743 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:03.743 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:03.743 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:03.744 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:03.744 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.744 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:03.744 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.065734+0000", 2024-04-19T02:04:03.744 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.744 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.744 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.744 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:03.744 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.065734+0000", 2024-04-19T02:04:03.744 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.744 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.744 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.744 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:03.744 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.065734+0000", 2024-04-19T02:04:03.744 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2399999999999999e-07 2024-04-19T02:04:03.744 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.745 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.745 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:03.745 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.065763+0000", 2024-04-19T02:04:03.745 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8393000000000001e-05 2024-04-19T02:04:03.745 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.745 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.745 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:03.745 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.065786+0000", 2024-04-19T02:04:03.745 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2909e-05 2024-04-19T02:04:03.745 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.745 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.745 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:03.745 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.065799+0000", 2024-04-19T02:04:03.745 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2975e-05 2024-04-19T02:04:03.745 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.746 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.746 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:03.746 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.069211+0000", 2024-04-19T02:04:03.746 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0034118899999999999 2024-04-19T02:04:03.746 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.746 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.746 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:03.746 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.069234+0000", 2024-04-19T02:04:03.746 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.372e-05 2024-04-19T02:04:03.746 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.746 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.746 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:03.746 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.074695+0000", 2024-04-19T02:04:03.746 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0054611779999999997 2024-04-19T02:04:03.746 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.747 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.747 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:03.747 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.074764+0000", 2024-04-19T02:04:03.747 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8954000000000004e-05 2024-04-19T02:04:03.747 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.747 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:03.747 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.747 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.747 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.747 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.as7 46/26 ECSubWrite(tid=103, reqid=osd.11.0:103, at_version=46'49, trim_to=0'0, roll_forward_to=45'45))", 2024-04-19T02:04:03.747 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.070195+0000", 2024-04-19T02:04:03.747 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 33.637984930000002, 2024-04-19T02:04:03.747 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0048631990000000003, 2024-04-19T02:04:03.747 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:03.747 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:03.748 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:03.748 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:03.748 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.748 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:03.748 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.070195+0000", 2024-04-19T02:04:03.748 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.748 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.748 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.748 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:03.748 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.070195+0000", 2024-04-19T02:04:03.748 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.748 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.748 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.748 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:03.748 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.070196+0000", 2024-04-19T02:04:03.748 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8899999999999999e-07 2024-04-19T02:04:03.749 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.749 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.749 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:03.749 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.070219+0000", 2024-04-19T02:04:03.749 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3485999999999998e-05 2024-04-19T02:04:03.749 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.749 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.749 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:03.749 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.070230+0000", 2024-04-19T02:04:03.749 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1039000000000001e-05 2024-04-19T02:04:03.749 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.749 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.749 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:03.749 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.070241+0000", 2024-04-19T02:04:03.750 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0866999999999999e-05 2024-04-19T02:04:03.750 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.750 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.750 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:03.750 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.070720+0000", 2024-04-19T02:04:03.750 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00047945099999999999 2024-04-19T02:04:03.750 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.750 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.750 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:03.750 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.070734+0000", 2024-04-19T02:04:03.750 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3811e-05 2024-04-19T02:04:03.750 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.750 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.750 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:03.750 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.075014+0000", 2024-04-19T02:04:03.751 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.004280196 2024-04-19T02:04:03.751 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.751 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.751 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:03.751 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.075058+0000", 2024-04-19T02:04:03.751 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.3659999999999999e-05 2024-04-19T02:04:03.751 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.751 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:03.751 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.751 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.751 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.751 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s5 46/26 ECSubWrite(tid=59, reqid=osd.1.0:58, at_version=0'0, trim_to=0'0, roll_forward_to=46'44))", 2024-04-19T02:04:03.751 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.077390+0000", 2024-04-19T02:04:03.751 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 33.630789425000003, 2024-04-19T02:04:03.751 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0043230480000000003, 2024-04-19T02:04:03.752 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:03.752 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:03.752 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:03.752 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:03.752 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.752 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:03.752 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077390+0000", 2024-04-19T02:04:03.752 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.752 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.752 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.752 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:03.752 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077390+0000", 2024-04-19T02:04:03.752 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.752 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.752 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.752 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:03.753 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077392+0000", 2024-04-19T02:04:03.753 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2270000000000001e-06 2024-04-19T02:04:03.753 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.753 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.753 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:03.753 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077409+0000", 2024-04-19T02:04:03.753 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7748999999999999e-05 2024-04-19T02:04:03.753 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.753 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.753 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:03.753 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077436+0000", 2024-04-19T02:04:03.753 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6451000000000001e-05 2024-04-19T02:04:03.753 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.753 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.753 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:03.754 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077502+0000", 2024-04-19T02:04:03.754 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.6493000000000002e-05 2024-04-19T02:04:03.754 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.754 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.754 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:03.754 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077638+0000", 2024-04-19T02:04:03.754 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013563599999999999 2024-04-19T02:04:03.754 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.754 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.754 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:03.754 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077650+0000", 2024-04-19T02:04:03.754 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1569999999999999e-05 2024-04-19T02:04:03.754 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.755 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.755 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:03.755 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.081644+0000", 2024-04-19T02:04:03.755 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0039942950000000001 2024-04-19T02:04:03.755 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.755 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.755 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:03.755 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.081713+0000", 2024-04-19T02:04:03.755 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9627000000000003e-05 2024-04-19T02:04:03.755 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.755 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:03.755 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.755 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.755 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.755 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es5 46/26 ECSubWrite(tid=39, reqid=client.4509.0:420, at_version=46'31, trim_to=0'0, roll_forward_to=46'29))", 2024-04-19T02:04:03.756 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.240512+0000", 2024-04-19T02:04:03.756 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 33.467667867000003, 2024-04-19T02:04:03.756 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0043966530000000004, 2024-04-19T02:04:03.756 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:03.756 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:03.756 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:03.756 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:03.756 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.756 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:03.756 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.240512+0000", 2024-04-19T02:04:03.756 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.756 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.757 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.757 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:03.757 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.240512+0000", 2024-04-19T02:04:03.757 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.757 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.757 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.757 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:03.757 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.240514+0000", 2024-04-19T02:04:03.757 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.717e-06 2024-04-19T02:04:03.757 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.757 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.757 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:03.757 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.242558+0000", 2024-04-19T02:04:03.757 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0020443940000000002 2024-04-19T02:04:03.757 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.758 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.758 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:03.758 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.242576+0000", 2024-04-19T02:04:03.758 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8127000000000001e-05 2024-04-19T02:04:03.758 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.758 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.758 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:03.758 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.242585+0000", 2024-04-19T02:04:03.758 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.1300000000000007e-06 2024-04-19T02:04:03.758 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.758 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.758 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:03.758 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.242678+0000", 2024-04-19T02:04:03.758 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.2808999999999999e-05 2024-04-19T02:04:03.758 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.759 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.759 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:03.759 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.242685+0000", 2024-04-19T02:04:03.759 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.7700000000000004e-06 2024-04-19T02:04:03.759 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.759 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.759 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:03.759 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.244876+0000", 2024-04-19T02:04:03.759 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0021906270000000001 2024-04-19T02:04:03.759 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.759 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.759 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:03.759 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.244909+0000", 2024-04-19T02:04:03.759 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3079e-05 2024-04-19T02:04:03.759 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.760 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:03.760 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.760 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.760 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.760 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s5 47/26 ECSubWrite(tid=62, reqid=client.4509.0:482, at_version=47'47, trim_to=0'0, roll_forward_to=47'45))", 2024-04-19T02:04:03.760 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:31.216102+0000", 2024-04-19T02:04:03.760 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 32.492078245999998, 2024-04-19T02:04:03.760 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0045766349999999999, 2024-04-19T02:04:03.760 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:03.760 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:03.760 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:03.760 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:03.760 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.760 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:03.760 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.216102+0000", 2024-04-19T02:04:03.760 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.761 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.761 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.761 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:03.761 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.216102+0000", 2024-04-19T02:04:03.761 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.761 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.761 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.761 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:03.761 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.216103+0000", 2024-04-19T02:04:03.761 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.683e-06 2024-04-19T02:04:03.761 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.761 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.761 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:03.761 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.216227+0000", 2024-04-19T02:04:03.761 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00012341500000000001 2024-04-19T02:04:03.762 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.762 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.762 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:03.762 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.216252+0000", 2024-04-19T02:04:03.762 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5525999999999999e-05 2024-04-19T02:04:03.762 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.762 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.762 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:03.762 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.216263+0000", 2024-04-19T02:04:03.762 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1273e-05 2024-04-19T02:04:03.762 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.762 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.762 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:03.762 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.216367+0000", 2024-04-19T02:04:03.763 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000103427 2024-04-19T02:04:03.763 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.763 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.763 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:03.763 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.216378+0000", 2024-04-19T02:04:03.763 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0684e-05 2024-04-19T02:04:03.763 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.763 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.763 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:03.763 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.220623+0000", 2024-04-19T02:04:03.763 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0042451870000000001 2024-04-19T02:04:03.763 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.763 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.763 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:03.763 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.220678+0000", 2024-04-19T02:04:03.764 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5439999999999998e-05 2024-04-19T02:04:03.764 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.764 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:03.764 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.764 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.764 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.764 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 51/26 ECSubWrite(tid=52, reqid=osd.9.0:51, at_version=0'0, trim_to=0'0, roll_forward_to=51'30))", 2024-04-19T02:04:03.764 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:35.116856+0000", 2024-04-19T02:04:03.764 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 28.591323739, 2024-04-19T02:04:03.764 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0055490599999999998, 2024-04-19T02:04:03.764 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:03.764 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:03.764 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:03.764 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:03.764 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.765 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:03.765 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:35.116856+0000", 2024-04-19T02:04:03.765 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.765 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.765 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.765 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:03.765 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:35.116856+0000", 2024-04-19T02:04:03.765 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.765 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.765 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.765 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:03.765 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:35.116857+0000", 2024-04-19T02:04:03.765 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0800000000000004e-07 2024-04-19T02:04:03.765 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.765 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.766 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:03.766 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:35.116871+0000", 2024-04-19T02:04:03.766 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.456e-05 2024-04-19T02:04:03.766 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.766 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.766 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:03.766 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:35.116884+0000", 2024-04-19T02:04:03.766 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2374999999999999e-05 2024-04-19T02:04:03.766 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.766 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.766 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:03.766 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:35.116894+0000", 2024-04-19T02:04:03.766 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.026e-05 2024-04-19T02:04:03.766 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.766 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.767 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:03.767 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:35.116986+0000", 2024-04-19T02:04:03.767 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.2021999999999994e-05 2024-04-19T02:04:03.767 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.767 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.767 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:03.767 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:35.117001+0000", 2024-04-19T02:04:03.767 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5235999999999999e-05 2024-04-19T02:04:03.767 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.767 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.767 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:03.767 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:35.122349+0000", 2024-04-19T02:04:03.767 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0053481400000000004 2024-04-19T02:04:03.767 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.767 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.768 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:03.768 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:35.122405+0000", 2024-04-19T02:04:03.768 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5658999999999997e-05 2024-04-19T02:04:03.768 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.768 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:03.768 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.768 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.768 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.768 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s5 57/26 ECSubWrite(tid=71, reqid=osd.1.0:71, at_version=57'56, trim_to=0'0, roll_forward_to=55'54))", 2024-04-19T02:04:03.768 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:41.141947+0000", 2024-04-19T02:04:03.768 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 22.566232352, 2024-04-19T02:04:03.768 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0043941919999999999, 2024-04-19T02:04:03.768 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:03.768 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:03.768 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:03.768 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:03.769 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.769 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:03.769 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.141947+0000", 2024-04-19T02:04:03.769 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.769 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.769 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.769 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:03.769 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.141947+0000", 2024-04-19T02:04:03.769 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.769 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.769 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.769 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:03.769 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.141948+0000", 2024-04-19T02:04:03.769 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.3700000000000001e-07 2024-04-19T02:04:03.769 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.770 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.770 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:03.770 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.141986+0000", 2024-04-19T02:04:03.770 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.7758e-05 2024-04-19T02:04:03.770 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.770 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.770 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:03.770 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.142020+0000", 2024-04-19T02:04:03.770 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4146999999999997e-05 2024-04-19T02:04:03.770 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.770 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.770 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:03.770 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.142036+0000", 2024-04-19T02:04:03.770 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5726000000000002e-05 2024-04-19T02:04:03.770 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.771 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.771 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:03.771 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.142223+0000", 2024-04-19T02:04:03.771 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00018726600000000001 2024-04-19T02:04:03.771 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.771 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.771 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:03.771 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.142241+0000", 2024-04-19T02:04:03.771 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8413000000000001e-05 2024-04-19T02:04:03.771 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.771 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.771 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:03.771 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.146291+0000", 2024-04-19T02:04:03.771 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0040499999999999998 2024-04-19T02:04:03.771 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.772 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.772 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:03.772 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.146342+0000", 2024-04-19T02:04:03.772 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.0445000000000002e-05 2024-04-19T02:04:03.772 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.772 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:03.772 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.772 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.772 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.772 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.as7 57/26 ECSubWrite(tid=136, reqid=osd.11.0:136, at_version=57'62, trim_to=0'0, roll_forward_to=55'60))", 2024-04-19T02:04:03.772 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:41.154457+0000", 2024-04-19T02:04:03.772 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 22.553722755999999, 2024-04-19T02:04:03.772 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0055113590000000004, 2024-04-19T02:04:03.772 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:03.772 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:03.772 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:03.773 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:03.773 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.773 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:03.773 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.154457+0000", 2024-04-19T02:04:03.773 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.773 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.773 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.773 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:03.773 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.154457+0000", 2024-04-19T02:04:03.773 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.773 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.773 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.773 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:03.773 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.154457+0000", 2024-04-19T02:04:03.773 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2300000000000002e-07 2024-04-19T02:04:03.774 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.774 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.774 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:03.774 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.154489+0000", 2024-04-19T02:04:03.774 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1301999999999999e-05 2024-04-19T02:04:03.774 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.774 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.774 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:03.774 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.154506+0000", 2024-04-19T02:04:03.774 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7554000000000002e-05 2024-04-19T02:04:03.774 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.774 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.774 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:03.774 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.154519+0000", 2024-04-19T02:04:03.774 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3074000000000001e-05 2024-04-19T02:04:03.775 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.775 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.775 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:03.775 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.154628+0000", 2024-04-19T02:04:03.775 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108974 2024-04-19T02:04:03.775 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.775 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.775 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:03.775 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.154653+0000", 2024-04-19T02:04:03.775 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4207999999999999e-05 2024-04-19T02:04:03.775 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.775 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.775 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:03.775 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.159911+0000", 2024-04-19T02:04:03.775 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0052585059999999996 2024-04-19T02:04:03.776 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.776 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.776 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:03.776 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.159968+0000", 2024-04-19T02:04:03.776 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7417999999999998e-05 2024-04-19T02:04:03.776 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.776 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:03.776 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.776 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.776 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.776 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s5 57/26 ECSubWrite(tid=72, reqid=osd.1.0:71, at_version=0'0, trim_to=0'0, roll_forward_to=57'56))", 2024-04-19T02:04:03.776 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:41.156487+0000", 2024-04-19T02:04:03.776 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 22.551692938999999, 2024-04-19T02:04:03.776 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0058785720000000003, 2024-04-19T02:04:03.776 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:03.776 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:03.777 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:03.777 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:03.777 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.777 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:03.777 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.156487+0000", 2024-04-19T02:04:03.777 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.777 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.777 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.777 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:03.777 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.156487+0000", 2024-04-19T02:04:03.777 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.777 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.777 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.777 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:03.777 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.156488+0000", 2024-04-19T02:04:03.778 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2449999999999999e-06 2024-04-19T02:04:03.778 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.778 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.778 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:03.778 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.156511+0000", 2024-04-19T02:04:03.778 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2429000000000002e-05 2024-04-19T02:04:03.778 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.778 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.778 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:03.778 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.156530+0000", 2024-04-19T02:04:03.778 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9389999999999999e-05 2024-04-19T02:04:03.778 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.778 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.778 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:03.779 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.156543+0000", 2024-04-19T02:04:03.779 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2676e-05 2024-04-19T02:04:03.779 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.779 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.779 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:03.779 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.156876+0000", 2024-04-19T02:04:03.779 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00033304599999999998 2024-04-19T02:04:03.779 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.779 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.779 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:03.779 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.156891+0000", 2024-04-19T02:04:03.779 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4921e-05 2024-04-19T02:04:03.779 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.779 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.779 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:03.780 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.162307+0000", 2024-04-19T02:04:03.780 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.005416227 2024-04-19T02:04:03.780 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.780 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.780 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:03.780 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.162365+0000", 2024-04-19T02:04:03.780 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.8638000000000003e-05 2024-04-19T02:04:03.780 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.780 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:03.780 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.780 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.780 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.780 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.5s2 62/26 ECSubWrite(tid=78, reqid=client.4509.0:703, at_version=62'35, trim_to=0'0, roll_forward_to=45'33))", 2024-04-19T02:04:03.780 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:46.470518+0000", 2024-04-19T02:04:03.780 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 17.237661430999999, 2024-04-19T02:04:03.781 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0042076220000000003, 2024-04-19T02:04:03.781 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:03.781 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:03.781 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:03.781 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:03.781 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.781 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:03.781 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.470518+0000", 2024-04-19T02:04:03.781 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.781 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.781 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.781 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:03.781 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.470518+0000", 2024-04-19T02:04:03.781 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.781 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.781 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.782 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:03.782 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.470519+0000", 2024-04-19T02:04:03.782 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2000000000000001e-07 2024-04-19T02:04:03.782 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.782 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.782 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:03.782 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.470690+0000", 2024-04-19T02:04:03.782 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00017127200000000001 2024-04-19T02:04:03.782 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.782 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.782 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:03.782 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.470714+0000", 2024-04-19T02:04:03.782 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.357e-05 2024-04-19T02:04:03.782 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.782 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.783 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:03.783 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.470726+0000", 2024-04-19T02:04:03.783 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2272e-05 2024-04-19T02:04:03.783 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.783 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.783 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:03.783 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.470859+0000", 2024-04-19T02:04:03.783 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013266999999999999 2024-04-19T02:04:03.783 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.783 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.783 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:03.783 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.470869+0000", 2024-04-19T02:04:03.783 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0838e-05 2024-04-19T02:04:03.783 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.783 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.784 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:03.784 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.474672+0000", 2024-04-19T02:04:03.784 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.003802911 2024-04-19T02:04:03.784 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.784 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.784 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:03.784 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.474726+0000", 2024-04-19T02:04:03.784 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.3769e-05 2024-04-19T02:04:03.784 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.784 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:03.784 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.784 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.784 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.784 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.5s2 67/26 ECSubWrite(tid=85, reqid=client.4509.0:765, at_version=67'41, trim_to=0'0, roll_forward_to=64'39))", 2024-04-19T02:04:03.784 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:51.097679+0000", 2024-04-19T02:04:03.784 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 12.610500395000001, 2024-04-19T02:04:03.785 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0042775929999999997, 2024-04-19T02:04:03.785 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:03.785 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:03.785 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:03.785 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:03.785 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.785 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:03.785 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:51.097679+0000", 2024-04-19T02:04:03.785 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.785 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.785 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.785 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:03.785 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:51.097679+0000", 2024-04-19T02:04:03.785 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.785 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.786 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.786 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:03.786 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:51.097680+0000", 2024-04-19T02:04:03.786 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.22e-07 2024-04-19T02:04:03.786 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.786 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.786 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:03.786 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:51.098288+0000", 2024-04-19T02:04:03.786 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00060858400000000001 2024-04-19T02:04:03.786 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.786 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.786 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:03.786 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:51.098304+0000", 2024-04-19T02:04:03.786 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6030999999999999e-05 2024-04-19T02:04:03.786 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.787 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.787 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:03.787 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:51.098315+0000", 2024-04-19T02:04:03.787 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1056000000000001e-05 2024-04-19T02:04:03.787 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.787 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.787 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:03.787 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:51.098435+0000", 2024-04-19T02:04:03.787 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000119743 2024-04-19T02:04:03.787 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.787 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.787 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:03.787 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:51.098449+0000", 2024-04-19T02:04:03.787 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3750999999999999e-05 2024-04-19T02:04:03.787 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.787 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.788 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:03.788 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:51.101907+0000", 2024-04-19T02:04:03.788 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.003458544 2024-04-19T02:04:03.788 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.788 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.788 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:03.788 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:51.101957+0000", 2024-04-19T02:04:03.788 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9561999999999997e-05 2024-04-19T02:04:03.788 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.788 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:03.788 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.788 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.788 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.788 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.as7 76/26 ECSubWrite(tid=194, reqid=client.4509.0:884, at_version=76'86, trim_to=0'0, roll_forward_to=73'84))", 2024-04-19T02:04:03.789 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:04:00.216904+0000", 2024-04-19T02:04:03.789 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 3.4912761539999999, 2024-04-19T02:04:03.789 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0086969809999999995, 2024-04-19T02:04:03.789 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:03.789 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:03.789 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:03.789 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:03.789 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.789 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:03.789 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.216904+0000", 2024-04-19T02:04:03.789 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.789 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.789 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.789 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:03.789 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.216904+0000", 2024-04-19T02:04:03.790 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:03.790 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.790 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.790 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:03.790 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.216905+0000", 2024-04-19T02:04:03.790 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6929999999999999e-06 2024-04-19T02:04:03.790 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.790 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.790 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:03.790 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.221110+0000", 2024-04-19T02:04:03.790 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.004204246 2024-04-19T02:04:03.790 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.790 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.790 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:03.791 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.221134+0000", 2024-04-19T02:04:03.791 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.425e-05 2024-04-19T02:04:03.791 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.791 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.791 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:03.791 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.221146+0000", 2024-04-19T02:04:03.791 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2463000000000001e-05 2024-04-19T02:04:03.791 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.791 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.791 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:03.791 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.221251+0000", 2024-04-19T02:04:03.791 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010493300000000001 2024-04-19T02:04:03.791 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.791 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.791 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:03.792 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.221268+0000", 2024-04-19T02:04:03.792 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6690000000000001e-05 2024-04-19T02:04:03.792 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.792 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.792 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:03.792 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.225562+0000", 2024-04-19T02:04:03.792 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0042940410000000002 2024-04-19T02:04:03.792 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:03.792 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:03.792 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:03.792 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.225601+0000", 2024-04-19T02:04:03.792 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.8665000000000003e-05 2024-04-19T02:04:03.792 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.792 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:03.792 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.793 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:03.793 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:03.793 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:04:03.794 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-19T02:04:03.795 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:04:03.793+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:04:03.895 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:04:03.897+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:04:03.925 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:04:03.925 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:04:03.925 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-04-19T02:04:03.925 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:04:03.939 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-19T02:04:03.996 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:04:03.997+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:04:04.060 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:04:04.060 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:04:04.060 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-04-19T02:04:04.060 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-04-19T02:04:04.060 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:04:04.074 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-19T02:04:04.115 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:04:04.117+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:04:04.148 INFO:tasks.rados.rados.0.smithi059.stdout:363: done (2 left) 2024-04-19T02:04:04.148 INFO:tasks.rados.rados.0.smithi059.stdout:364: snap_create 2024-04-19T02:04:04.148 INFO:tasks.rados.rados.0.smithi059.stdout:361: finishing copy_from racing read to smithi05930959-47 2024-04-19T02:04:04.148 INFO:tasks.rados.rados.0.smithi059.stdout:362: finishing write tid 1 to smithi05930959-21 2024-04-19T02:04:04.148 INFO:tasks.rados.rados.0.smithi059.stdout:362: finishing write tid 2 to smithi05930959-21 2024-04-19T02:04:04.148 INFO:tasks.rados.rados.0.smithi059.stdout:361: finishing copy_from to smithi05930959-47 2024-04-19T02:04:04.148 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 47 v 64 (ObjNum 94 snap 9 seq_num 94) dirty exists 2024-04-19T02:04:04.148 INFO:tasks.rados.rados.0.smithi059.stdout:362: finishing write tid 3 to smithi05930959-21 2024-04-19T02:04:04.149 INFO:tasks.rados.rados.0.smithi059.stdout:362: oid 21 updating version 0 to 174 2024-04-19T02:04:04.149 INFO:tasks.rados.rados.0.smithi059.stdout:362: oid 21 version 174 is already newer than 173 2024-04-19T02:04:04.149 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 21 v 174 (ObjNum 158 snap 29 seq_num 158) dirty exists 2024-04-19T02:04:04.149 INFO:tasks.rados.rados.0.smithi059.stdout:362: left oid 21 (ObjNum 158 snap 29 seq_num 158) 2024-04-19T02:04:04.199 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:04:04.201+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:04:04.270 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:04:04.270 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-04-19T02:04:04.270 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-04-19T02:04:04.270 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-04-19T02:04:04.270 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.270 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:238 3.2s0 3:407ecba7:::smithi05930959-22:2 [read 0~360448,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-19T02:04:04.270 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.646510+0000", 2024-04-19T02:04:04.271 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 47.624987902000001, 2024-04-19T02:04:04.271 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.028744658999999999, 2024-04-19T02:04:04.271 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:04.271 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:04.271 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:04:04.271 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:04.271 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:04.271 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.271 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 238 2024-04-19T02:04:04.271 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.271 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:04.271 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.271 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:04.271 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646510+0000", 2024-04-19T02:04:04.271 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.272 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.272 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.272 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:04.272 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646510+0000", 2024-04-19T02:04:04.272 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.272 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.272 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.272 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:04.272 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646511+0000", 2024-04-19T02:04:04.272 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.8599999999999997e-07 2024-04-19T02:04:04.272 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.272 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.272 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:04.273 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646517+0000", 2024-04-19T02:04:04.273 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.1700000000000002e-06 2024-04-19T02:04:04.273 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.273 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.273 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:04.273 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646528+0000", 2024-04-19T02:04:04.273 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1202999999999999e-05 2024-04-19T02:04:04.273 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.273 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.273 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.273 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646537+0000", 2024-04-19T02:04:04.273 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7460000000000006e-06 2024-04-19T02:04:04.273 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.273 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.274 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:04.274 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.647105+0000", 2024-04-19T02:04:04.274 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00056801599999999996 2024-04-19T02:04:04.274 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.274 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.274 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:04:04.274 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.647127+0000", 2024-04-19T02:04:04.274 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.175e-05 2024-04-19T02:04:04.274 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.274 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.274 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:04.274 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.657504+0000", 2024-04-19T02:04:04.274 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010377961 2024-04-19T02:04:04.274 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.275 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.275 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:04.275 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.657826+0000", 2024-04-19T02:04:04.275 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00032128300000000001 2024-04-19T02:04:04.275 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.275 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.275 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:04.275 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.675254+0000", 2024-04-19T02:04:04.275 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.017428644 2024-04-19T02:04:04.275 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.275 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:04.275 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.275 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.275 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.275 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:239 3.2s0 3:407ecba7:::smithi05930959-22:2 [read 0~360448,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-19T02:04:04.276 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.646549+0000", 2024-04-19T02:04:04.276 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 47.624948431, 2024-04-19T02:04:04.276 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.029381570999999999, 2024-04-19T02:04:04.276 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:04.276 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:04.276 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:04:04.276 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:04.276 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:04.276 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.276 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 239 2024-04-19T02:04:04.276 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.276 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:04.276 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.276 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:04.276 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646549+0000", 2024-04-19T02:04:04.277 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.277 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.277 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.277 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:04.277 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646549+0000", 2024-04-19T02:04:04.277 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.277 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.277 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.277 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:04.277 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646550+0000", 2024-04-19T02:04:04.277 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8400000000000004e-07 2024-04-19T02:04:04.277 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.277 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.277 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:04.277 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646555+0000", 2024-04-19T02:04:04.278 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9989999999999999e-06 2024-04-19T02:04:04.278 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.278 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.278 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:04.278 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646567+0000", 2024-04-19T02:04:04.278 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.202e-05 2024-04-19T02:04:04.278 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.278 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.278 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.278 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646575+0000", 2024-04-19T02:04:04.278 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.7640000000000005e-06 2024-04-19T02:04:04.278 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.278 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.278 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:04.278 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.647229+0000", 2024-04-19T02:04:04.279 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00065415299999999998 2024-04-19T02:04:04.279 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.279 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.279 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:04:04.279 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.647255+0000", 2024-04-19T02:04:04.279 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5795999999999998e-05 2024-04-19T02:04:04.279 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.279 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.279 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:04.279 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.659835+0000", 2024-04-19T02:04:04.279 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012580713 2024-04-19T02:04:04.279 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.279 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.279 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:04.279 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.660090+0000", 2024-04-19T02:04:04.280 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00025430899999999998 2024-04-19T02:04:04.280 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.280 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.280 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:04.280 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.675931+0000", 2024-04-19T02:04:04.280 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.015841133 2024-04-19T02:04:04.280 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.280 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:04.280 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.280 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.280 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.280 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:349 3.2s0 3:43b7722e:::smithi05930959-10:head [copy-from ver 11] snapc 8=[8,6,5] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:04:04.280 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.746333+0000", 2024-04-19T02:04:04.281 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 39.525164273999998, 2024-04-19T02:04:04.281 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.034219922999999999, 2024-04-19T02:04:04.281 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:04.281 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:04.281 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:04.281 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:04.281 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:04.281 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.281 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 349 2024-04-19T02:04:04.281 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.281 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:04.281 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.281 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:04.282 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746333+0000", 2024-04-19T02:04:04.282 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.282 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.282 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.282 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:04.282 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746333+0000", 2024-04-19T02:04:04.282 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.282 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.282 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.282 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:04.282 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746337+0000", 2024-04-19T02:04:04.282 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4769999999999999e-06 2024-04-19T02:04:04.282 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.282 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.282 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:04.283 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746352+0000", 2024-04-19T02:04:04.283 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5277e-05 2024-04-19T02:04:04.283 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.283 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.283 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:04.283 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746387+0000", 2024-04-19T02:04:04.283 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4436e-05 2024-04-19T02:04:04.283 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.283 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.283 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.283 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746403+0000", 2024-04-19T02:04:04.283 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6475999999999999e-05 2024-04-19T02:04:04.283 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.283 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.284 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:04.284 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746490+0000", 2024-04-19T02:04:04.284 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7213000000000007e-05 2024-04-19T02:04:04.284 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.284 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.284 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "op must wait for map", 2024-04-19T02:04:04.284 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746497+0000", 2024-04-19T02:04:04.284 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.0520000000000004e-06 2024-04-19T02:04:04.284 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.284 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.284 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:04.284 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749865+0000", 2024-04-19T02:04:04.284 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0033676069999999999 2024-04-19T02:04:04.284 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.285 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.285 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:04.285 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749938+0000", 2024-04-19T02:04:04.285 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.3530999999999998e-05 2024-04-19T02:04:04.285 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.285 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.285 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:04.285 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.768855+0000", 2024-04-19T02:04:04.285 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.018916493 2024-04-19T02:04:04.285 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.285 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.285 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:04.285 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.771377+0000", 2024-04-19T02:04:04.285 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0025221520000000002 2024-04-19T02:04:04.285 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.286 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.286 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:04.286 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.780460+0000", 2024-04-19T02:04:04.286 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0090830389999999993 2024-04-19T02:04:04.286 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.286 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.286 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:04.286 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.780553+0000", 2024-04-19T02:04:04.286 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3170000000000004e-05 2024-04-19T02:04:04.286 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.286 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:04.286 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.286 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.286 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.287 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:350 3.2s0 3:43b7722e:::smithi05930959-10:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:04:04.287 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.746450+0000", 2024-04-19T02:04:04.287 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 39.525047614999998, 2024-04-19T02:04:04.287 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.024575331999999998, 2024-04-19T02:04:04.287 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:04.287 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:04.287 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:04:04.287 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:04.287 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:04.287 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.287 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 350 2024-04-19T02:04:04.287 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.287 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:04.287 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.288 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:04.288 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746450+0000", 2024-04-19T02:04:04.288 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.288 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.288 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.288 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:04.288 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746450+0000", 2024-04-19T02:04:04.288 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.288 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.288 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.288 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:04.288 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746451+0000", 2024-04-19T02:04:04.288 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0400000000000005e-07 2024-04-19T02:04:04.288 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.289 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.289 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:04.289 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746456+0000", 2024-04-19T02:04:04.289 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5199999999999997e-06 2024-04-19T02:04:04.289 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.289 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.289 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:04.289 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746470+0000", 2024-04-19T02:04:04.289 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3935e-05 2024-04-19T02:04:04.289 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.289 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.289 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.289 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746481+0000", 2024-04-19T02:04:04.289 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0919e-05 2024-04-19T02:04:04.289 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.290 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.290 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:04.290 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746552+0000", 2024-04-19T02:04:04.290 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.0235999999999996e-05 2024-04-19T02:04:04.290 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.290 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.290 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:04:04.290 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746557+0000", 2024-04-19T02:04:04.290 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.6570000000000002e-06 2024-04-19T02:04:04.290 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.290 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.290 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:04.290 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750143+0000", 2024-04-19T02:04:04.291 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0035860549999999999 2024-04-19T02:04:04.291 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.291 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.291 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for blocked object", 2024-04-19T02:04:04.291 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750177+0000", 2024-04-19T02:04:04.291 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3796e-05 2024-04-19T02:04:04.291 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.291 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.291 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:04.291 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.770842+0000", 2024-04-19T02:04:04.291 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.020664657999999999 2024-04-19T02:04:04.291 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.291 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.291 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:04.291 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.770923+0000", 2024-04-19T02:04:04.292 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0869000000000001e-05 2024-04-19T02:04:04.292 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.292 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.292 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:04.292 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.771025+0000", 2024-04-19T02:04:04.292 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010278299999999999 2024-04-19T02:04:04.292 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.292 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:04.292 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.292 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.292 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.292 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:422 3.2s0 3:43b7722e:::smithi05930959-10:head [copy-from ver 11] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:04.292 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.193952+0000", 2024-04-19T02:04:04.293 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 34.077545276000002, 2024-04-19T02:04:04.293 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.044114190999999997, 2024-04-19T02:04:04.293 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:04.293 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:04.293 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:04.293 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:04.293 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:04.293 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.293 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 422 2024-04-19T02:04:04.293 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.293 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:04.293 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.293 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:04.294 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.193952+0000", 2024-04-19T02:04:04.294 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.294 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.294 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.294 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:04.294 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.193952+0000", 2024-04-19T02:04:04.294 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.294 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.294 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.294 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:04.294 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.193955+0000", 2024-04-19T02:04:04.294 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3159999999999999e-06 2024-04-19T02:04:04.294 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.294 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.295 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:04.295 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.193968+0000", 2024-04-19T02:04:04.295 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3726999999999999e-05 2024-04-19T02:04:04.295 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.295 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.295 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:04.295 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.193993+0000", 2024-04-19T02:04:04.295 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4499e-05 2024-04-19T02:04:04.295 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.295 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.295 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.295 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194005+0000", 2024-04-19T02:04:04.295 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1770000000000001e-05 2024-04-19T02:04:04.295 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.296 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.296 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:04.296 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194109+0000", 2024-04-19T02:04:04.296 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000103904 2024-04-19T02:04:04.296 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.296 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.296 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:04.296 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194249+0000", 2024-04-19T02:04:04.296 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014036699999999999 2024-04-19T02:04:04.296 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.296 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.296 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:04.296 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.224308+0000", 2024-04-19T02:04:04.296 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.030059210999999999 2024-04-19T02:04:04.297 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.297 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.297 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:04.297 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.237172+0000", 2024-04-19T02:04:04.297 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012863734999999999 2024-04-19T02:04:04.297 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.297 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.297 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:04.297 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.237230+0000", 2024-04-19T02:04:04.297 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.8238e-05 2024-04-19T02:04:04.297 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.297 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.297 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:04.298 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.238067+0000", 2024-04-19T02:04:04.298 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000836424 2024-04-19T02:04:04.298 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.298 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:04.298 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.298 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.298 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.298 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:423 3.2s0 3:43b7722e:::smithi05930959-10:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:04.298 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194040+0000", 2024-04-19T02:04:04.298 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 34.077457946000003, 2024-04-19T02:04:04.298 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.040466800999999997, 2024-04-19T02:04:04.298 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:04.298 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:04.298 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:04:04.299 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:04.299 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:04.299 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.299 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 423 2024-04-19T02:04:04.299 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.299 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:04.299 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.299 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:04.299 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194040+0000", 2024-04-19T02:04:04.299 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.299 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.299 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.299 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:04.299 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194040+0000", 2024-04-19T02:04:04.300 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.300 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.300 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.300 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:04.300 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194040+0000", 2024-04-19T02:04:04.300 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.4700000000000001e-07 2024-04-19T02:04:04.300 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.300 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.300 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:04.300 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194044+0000", 2024-04-19T02:04:04.300 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9049999999999999e-06 2024-04-19T02:04:04.300 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.300 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.300 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:04.301 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194056+0000", 2024-04-19T02:04:04.301 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1567e-05 2024-04-19T02:04:04.301 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.301 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.301 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.301 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194063+0000", 2024-04-19T02:04:04.301 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.1539999999999996e-06 2024-04-19T02:04:04.301 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.301 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.301 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:04.301 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194511+0000", 2024-04-19T02:04:04.301 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00044832999999999999 2024-04-19T02:04:04.301 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.301 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.301 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for blocked object", 2024-04-19T02:04:04.302 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194561+0000", 2024-04-19T02:04:04.302 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9336999999999999e-05 2024-04-19T02:04:04.302 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.302 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.302 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:04.302 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.234276+0000", 2024-04-19T02:04:04.302 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.039715156000000001 2024-04-19T02:04:04.302 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.302 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.302 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:04.302 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.234405+0000", 2024-04-19T02:04:04.302 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00012947100000000001 2024-04-19T02:04:04.302 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.303 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.303 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:04.303 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.234507+0000", 2024-04-19T02:04:04.303 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000101234 2024-04-19T02:04:04.303 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.303 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:04.303 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.303 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.303 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.303 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:424 3.2s0 3:407ecba7:::smithi05930959-22:head [append 0~0] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:04.303 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194090+0000", 2024-04-19T02:04:04.303 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 34.077407201, 2024-04-19T02:04:04.303 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.04103888, 2024-04-19T02:04:04.304 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:04.304 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:04.304 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:04.304 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:04.304 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:04.304 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.304 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 424 2024-04-19T02:04:04.304 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.304 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:04.304 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.304 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:04.304 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194090+0000", 2024-04-19T02:04:04.304 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.304 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.304 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.304 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:04.305 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194090+0000", 2024-04-19T02:04:04.305 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.305 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.305 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.305 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:04.305 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194091+0000", 2024-04-19T02:04:04.305 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.9599999999999999e-07 2024-04-19T02:04:04.305 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.305 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.305 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:04.305 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194095+0000", 2024-04-19T02:04:04.305 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.019e-06 2024-04-19T02:04:04.305 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.305 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.306 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:04.306 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194104+0000", 2024-04-19T02:04:04.306 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.6030000000000005e-06 2024-04-19T02:04:04.306 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.306 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.306 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.306 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194110+0000", 2024-04-19T02:04:04.306 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.6880000000000002e-06 2024-04-19T02:04:04.306 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.306 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.306 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:04.306 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194597+0000", 2024-04-19T02:04:04.306 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00048649499999999998 2024-04-19T02:04:04.306 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.307 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.307 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:04.307 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194669+0000", 2024-04-19T02:04:04.307 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.2081000000000003e-05 2024-04-19T02:04:04.307 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.307 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.307 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:04.307 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195298+0000", 2024-04-19T02:04:04.307 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000628635 2024-04-19T02:04:04.307 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.307 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.307 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:04.307 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.201226+0000", 2024-04-19T02:04:04.307 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0059289199999999998 2024-04-19T02:04:04.307 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.308 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.308 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:04.308 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235058+0000", 2024-04-19T02:04:04.308 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.033831539000000001 2024-04-19T02:04:04.308 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.308 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.308 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:04.308 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235129+0000", 2024-04-19T02:04:04.308 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.1303999999999994e-05 2024-04-19T02:04:04.308 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.308 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:04.308 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.308 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.308 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.309 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:425 3.2s0 3:407ecba7:::smithi05930959-22:head [rollback a] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:04.309 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194118+0000", 2024-04-19T02:04:04.309 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 34.077379737999998, 2024-04-19T02:04:04.309 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.041424615999999997, 2024-04-19T02:04:04.309 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:04.309 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:04.309 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:04.309 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:04.309 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:04.309 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.309 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 425 2024-04-19T02:04:04.309 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.309 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:04.309 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.310 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:04.310 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194118+0000", 2024-04-19T02:04:04.310 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.310 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.310 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.310 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:04.310 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194118+0000", 2024-04-19T02:04:04.310 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.310 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.310 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.310 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:04.310 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194118+0000", 2024-04-19T02:04:04.310 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7799999999999997e-07 2024-04-19T02:04:04.310 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.311 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.311 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:04.311 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194120+0000", 2024-04-19T02:04:04.311 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9740000000000001e-06 2024-04-19T02:04:04.311 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.311 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.311 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:04.311 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194124+0000", 2024-04-19T02:04:04.311 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1540000000000004e-06 2024-04-19T02:04:04.311 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.311 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.311 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.311 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194127+0000", 2024-04-19T02:04:04.311 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1379999999999999e-06 2024-04-19T02:04:04.312 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.312 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.312 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:04.312 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195791+0000", 2024-04-19T02:04:04.312 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0016630779999999999 2024-04-19T02:04:04.312 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.312 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.312 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:04.312 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195884+0000", 2024-04-19T02:04:04.312 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3885e-05 2024-04-19T02:04:04.312 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.312 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.312 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:04.312 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.196827+0000", 2024-04-19T02:04:04.313 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00094285400000000002 2024-04-19T02:04:04.313 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.313 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.313 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:04.313 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.201229+0000", 2024-04-19T02:04:04.313 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0044017359999999998 2024-04-19T02:04:04.313 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.313 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.313 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:04.313 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235479+0000", 2024-04-19T02:04:04.313 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.034249891999999997 2024-04-19T02:04:04.313 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.313 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.313 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:04.313 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235543+0000", 2024-04-19T02:04:04.314 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.3627000000000006e-05 2024-04-19T02:04:04.314 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.314 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:04.314 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.314 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.314 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.314 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:426 3.2s0 3:407ecba7:::smithi05930959-22:head [append 0~0] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:04.314 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194132+0000", 2024-04-19T02:04:04.314 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 34.077366126999998, 2024-04-19T02:04:04.314 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.041767809000000003, 2024-04-19T02:04:04.314 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:04.314 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:04.314 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:04.314 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:04.315 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:04.315 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.315 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 426 2024-04-19T02:04:04.315 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.315 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:04.315 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.315 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:04.315 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194132+0000", 2024-04-19T02:04:04.315 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.315 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.315 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.315 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:04.315 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194132+0000", 2024-04-19T02:04:04.315 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.316 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.316 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.316 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:04.316 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194132+0000", 2024-04-19T02:04:04.316 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5199999999999998e-07 2024-04-19T02:04:04.316 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.316 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.316 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:04.316 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194133+0000", 2024-04-19T02:04:04.316 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.624e-06 2024-04-19T02:04:04.316 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.316 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.316 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:04.317 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194137+0000", 2024-04-19T02:04:04.317 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3550000000000001e-06 2024-04-19T02:04:04.317 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.317 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.317 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.317 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194140+0000", 2024-04-19T02:04:04.317 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2229999999999999e-06 2024-04-19T02:04:04.317 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.317 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.317 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:04.317 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.198324+0000", 2024-04-19T02:04:04.317 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0041834799999999998 2024-04-19T02:04:04.317 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.317 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.318 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:04.318 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.198430+0000", 2024-04-19T02:04:04.318 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010597100000000001 2024-04-19T02:04:04.318 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.318 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.318 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:04.318 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.199339+0000", 2024-04-19T02:04:04.318 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00090987100000000003 2024-04-19T02:04:04.318 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.318 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.318 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:04.318 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.201230+0000", 2024-04-19T02:04:04.318 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0018903170000000001 2024-04-19T02:04:04.318 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.319 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.319 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:04.319 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235836+0000", 2024-04-19T02:04:04.319 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.034606382999999998 2024-04-19T02:04:04.319 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.319 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.319 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:04.319 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235899+0000", 2024-04-19T02:04:04.319 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.3232999999999996e-05 2024-04-19T02:04:04.319 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.319 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:04.319 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.319 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.319 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.319 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:437 3.bs0 3:d6e3a590:::smithi05930959-43:head [copy-from ver 51] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:04.320 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194173+0000", 2024-04-19T02:04:04.320 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 34.077324881000003, 2024-04-19T02:04:04.320 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.054675901999999998, 2024-04-19T02:04:04.320 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:04.320 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:04.320 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:04.320 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:04.320 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:04.320 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.320 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 437 2024-04-19T02:04:04.320 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.320 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:04.320 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.320 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:04.320 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194173+0000", 2024-04-19T02:04:04.321 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.321 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.321 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.321 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:04.321 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194173+0000", 2024-04-19T02:04:04.321 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.321 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.321 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.321 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:04.321 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194173+0000", 2024-04-19T02:04:04.321 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1100000000000002e-07 2024-04-19T02:04:04.321 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.321 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.321 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:04.321 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194179+0000", 2024-04-19T02:04:04.322 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.122e-06 2024-04-19T02:04:04.322 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.322 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.322 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:04.322 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194185+0000", 2024-04-19T02:04:04.322 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.3109999999999996e-06 2024-04-19T02:04:04.322 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.322 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.322 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.322 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194188+0000", 2024-04-19T02:04:04.322 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0230000000000001e-06 2024-04-19T02:04:04.322 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.322 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.322 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:04.323 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194246+0000", 2024-04-19T02:04:04.323 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.8865e-05 2024-04-19T02:04:04.323 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.323 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.323 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:04.323 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194356+0000", 2024-04-19T02:04:04.323 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000109558 2024-04-19T02:04:04.323 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.323 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.323 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:04.323 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.240077+0000", 2024-04-19T02:04:04.323 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.045721108000000003 2024-04-19T02:04:04.323 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.324 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.324 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:04.324 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.244667+0000", 2024-04-19T02:04:04.324 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0045899000000000001 2024-04-19T02:04:04.324 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.324 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.324 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:04.324 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.248766+0000", 2024-04-19T02:04:04.324 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0040986579999999998 2024-04-19T02:04:04.324 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.324 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.324 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:04.324 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.248849+0000", 2024-04-19T02:04:04.324 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3046000000000004e-05 2024-04-19T02:04:04.325 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.325 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:04.325 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.325 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.325 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.325 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:438 3.bs0 3:d6e3a590:::smithi05930959-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:04.325 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194196+0000", 2024-04-19T02:04:04.325 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 34.077302068000002, 2024-04-19T02:04:04.325 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.047992689999999998, 2024-04-19T02:04:04.325 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:04.325 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:04.325 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:04:04.325 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:04.325 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:04.326 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.326 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 438 2024-04-19T02:04:04.326 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.326 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:04.326 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.326 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:04.326 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194196+0000", 2024-04-19T02:04:04.326 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.326 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.326 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.326 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:04.326 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194196+0000", 2024-04-19T02:04:04.326 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.326 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.327 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.327 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:04.327 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194196+0000", 2024-04-19T02:04:04.327 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4700000000000002e-07 2024-04-19T02:04:04.327 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.327 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.327 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:04.327 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194198+0000", 2024-04-19T02:04:04.327 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4320000000000002e-06 2024-04-19T02:04:04.327 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.327 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.327 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:04.327 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194204+0000", 2024-04-19T02:04:04.327 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0850000000000002e-06 2024-04-19T02:04:04.328 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.328 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.328 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.328 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194209+0000", 2024-04-19T02:04:04.328 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5090000000000004e-06 2024-04-19T02:04:04.328 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.328 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.328 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:04.328 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194533+0000", 2024-04-19T02:04:04.328 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00032412499999999998 2024-04-19T02:04:04.328 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.328 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.328 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for blocked object", 2024-04-19T02:04:04.328 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194593+0000", 2024-04-19T02:04:04.329 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.9854e-05 2024-04-19T02:04:04.329 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.329 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.329 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:04.329 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.242048+0000", 2024-04-19T02:04:04.329 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.047455152 2024-04-19T02:04:04.329 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.329 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.329 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:04.329 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.242113+0000", 2024-04-19T02:04:04.329 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.4573000000000007e-05 2024-04-19T02:04:04.329 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.329 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.329 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:04.330 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.242188+0000", 2024-04-19T02:04:04.330 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.5612999999999996e-05 2024-04-19T02:04:04.330 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.330 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:04.330 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.330 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.330 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.330 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:439 3.2s0 3:43ce42f3:::smithi05930959-27:head [append 0~0] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:04.330 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194231+0000", 2024-04-19T02:04:04.330 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 34.077266635000001, 2024-04-19T02:04:04.330 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.041997487, 2024-04-19T02:04:04.330 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:04.331 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:04.331 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:04.331 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:04.331 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:04.331 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.331 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 439 2024-04-19T02:04:04.331 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.331 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:04.331 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.331 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:04.331 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194231+0000", 2024-04-19T02:04:04.331 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.331 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.331 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.332 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:04.332 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194231+0000", 2024-04-19T02:04:04.332 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.332 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.332 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.332 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:04.332 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194232+0000", 2024-04-19T02:04:04.332 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7999999999999995e-07 2024-04-19T02:04:04.332 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.332 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.332 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:04.332 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194236+0000", 2024-04-19T02:04:04.332 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2139999999999998e-06 2024-04-19T02:04:04.332 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.333 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.333 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:04.333 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194243+0000", 2024-04-19T02:04:04.333 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.5979999999999999e-06 2024-04-19T02:04:04.333 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.333 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.333 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.333 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194248+0000", 2024-04-19T02:04:04.333 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.3370000000000001e-06 2024-04-19T02:04:04.333 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.333 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.333 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:04.333 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.204081+0000", 2024-04-19T02:04:04.333 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0098332009999999997 2024-04-19T02:04:04.334 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.334 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.334 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:04.334 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.204143+0000", 2024-04-19T02:04:04.334 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.156e-05 2024-04-19T02:04:04.334 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.334 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.334 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:04.334 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.204875+0000", 2024-04-19T02:04:04.334 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00073241099999999998 2024-04-19T02:04:04.334 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.334 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.334 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:04.334 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.208933+0000", 2024-04-19T02:04:04.335 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0040575990000000003 2024-04-19T02:04:04.335 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.335 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.335 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:04.335 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.236159+0000", 2024-04-19T02:04:04.335 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.027226274000000002 2024-04-19T02:04:04.335 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.335 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.335 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:04.335 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.236229+0000", 2024-04-19T02:04:04.335 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9713000000000002e-05 2024-04-19T02:04:04.335 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.335 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:04.335 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.336 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.336 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.336 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:440 3.2s0 3:43ce42f3:::smithi05930959-27:head [rollback a] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:04.336 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194253+0000", 2024-04-19T02:04:04.336 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 34.077244395000001, 2024-04-19T02:04:04.336 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.042233729999999997, 2024-04-19T02:04:04.336 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:04.336 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:04.336 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:04.336 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:04.336 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:04.336 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.336 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 440 2024-04-19T02:04:04.336 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.336 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:04.337 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.337 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:04.337 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194253+0000", 2024-04-19T02:04:04.337 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.337 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.337 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.337 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:04.337 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194253+0000", 2024-04-19T02:04:04.337 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.337 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.337 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.337 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:04.337 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194254+0000", 2024-04-19T02:04:04.337 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8299999999999998e-07 2024-04-19T02:04:04.337 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.338 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.338 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:04.338 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194255+0000", 2024-04-19T02:04:04.338 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7120000000000001e-06 2024-04-19T02:04:04.338 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.338 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.338 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:04.338 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194259+0000", 2024-04-19T02:04:04.338 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.517e-06 2024-04-19T02:04:04.338 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.338 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.338 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.338 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194261+0000", 2024-04-19T02:04:04.339 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.689e-06 2024-04-19T02:04:04.339 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.339 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.339 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:04.339 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.205980+0000", 2024-04-19T02:04:04.339 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.011718561000000001 2024-04-19T02:04:04.339 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.339 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.339 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:04.339 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.206074+0000", 2024-04-19T02:04:04.339 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3556999999999999e-05 2024-04-19T02:04:04.339 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.339 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.339 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:04.340 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.206673+0000", 2024-04-19T02:04:04.340 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00059902100000000004 2024-04-19T02:04:04.340 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.340 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.340 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:04.340 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.208936+0000", 2024-04-19T02:04:04.340 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002262975 2024-04-19T02:04:04.340 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.340 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.340 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:04.340 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.236430+0000", 2024-04-19T02:04:04.340 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.027494338 2024-04-19T02:04:04.340 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.340 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.341 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:04.341 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.236487+0000", 2024-04-19T02:04:04.341 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7077000000000003e-05 2024-04-19T02:04:04.341 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.341 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:04.341 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.341 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.341 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.341 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:441 3.2s0 3:43ce42f3:::smithi05930959-27:head [append 0~0] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:04.341 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194270+0000", 2024-04-19T02:04:04.341 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 34.077227913000002, 2024-04-19T02:04:04.341 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.042541760999999997, 2024-04-19T02:04:04.341 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:04.341 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:04.342 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:04.342 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:04.342 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:04.342 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.342 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 441 2024-04-19T02:04:04.342 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.342 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:04.342 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.342 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:04.342 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194270+0000", 2024-04-19T02:04:04.342 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.342 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.342 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.342 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:04.343 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194270+0000", 2024-04-19T02:04:04.343 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.343 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.343 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.343 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:04.343 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194270+0000", 2024-04-19T02:04:04.343 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2500000000000001e-07 2024-04-19T02:04:04.343 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.343 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.343 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:04.343 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194271+0000", 2024-04-19T02:04:04.343 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3939999999999999e-06 2024-04-19T02:04:04.343 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.343 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.343 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:04.344 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194275+0000", 2024-04-19T02:04:04.344 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.827e-06 2024-04-19T02:04:04.344 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.344 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.344 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.344 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194278+0000", 2024-04-19T02:04:04.344 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7889999999999999e-06 2024-04-19T02:04:04.344 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.344 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.344 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:04.344 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.207160+0000", 2024-04-19T02:04:04.344 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012881867999999999 2024-04-19T02:04:04.344 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.344 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.344 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:04.345 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.207262+0000", 2024-04-19T02:04:04.345 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000102142 2024-04-19T02:04:04.345 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.345 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.345 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:04.345 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.207787+0000", 2024-04-19T02:04:04.345 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00052537099999999995 2024-04-19T02:04:04.345 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.345 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.345 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:04.345 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.208937+0000", 2024-04-19T02:04:04.345 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00114908 2024-04-19T02:04:04.345 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.345 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.346 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:04.346 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.236712+0000", 2024-04-19T02:04:04.346 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.027775428000000001 2024-04-19T02:04:04.346 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.346 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.346 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:04.346 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.236811+0000", 2024-04-19T02:04:04.346 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9537000000000006e-05 2024-04-19T02:04:04.346 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.346 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:04.346 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.346 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.346 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.346 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(osd.7.18:2 3.2s0 3:465821b6:::smithi05930959-32:head [assert-version v14,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:04.347 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.199189+0000", 2024-04-19T02:04:04.347 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 34.072308677999999, 2024-04-19T02:04:04.347 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.035632812, 2024-04-19T02:04:04.347 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:04.347 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:04.347 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:04:04.347 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:04.347 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.347 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:04.347 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.199189+0000", 2024-04-19T02:04:04.347 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.347 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.347 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.348 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:04.348 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.199189+0000", 2024-04-19T02:04:04.348 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.348 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.348 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.348 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:04.348 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.199190+0000", 2024-04-19T02:04:04.348 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5069999999999999e-06 2024-04-19T02:04:04.348 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.348 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.348 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:04.348 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.199198+0000", 2024-04-19T02:04:04.348 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.199e-06 2024-04-19T02:04:04.348 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.349 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.349 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:04.349 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.199211+0000", 2024-04-19T02:04:04.349 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3567000000000001e-05 2024-04-19T02:04:04.349 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.349 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.349 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.349 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.199220+0000", 2024-04-19T02:04:04.349 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2719999999999997e-06 2024-04-19T02:04:04.349 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.349 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.349 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:04.349 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.200485+0000", 2024-04-19T02:04:04.349 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001265195 2024-04-19T02:04:04.350 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.350 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.350 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:04.350 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.200596+0000", 2024-04-19T02:04:04.350 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00011098 2024-04-19T02:04:04.350 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.350 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.350 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:04.350 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.234822+0000", 2024-04-19T02:04:04.350 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.034226092 2024-04-19T02:04:04.350 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.350 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:04.350 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.350 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.350 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.351 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.2s0 46/26 ECSubReadReply(tid=166, attrs_read=0))", 2024-04-19T02:04:04.351 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.209587+0000", 2024-04-19T02:04:04.351 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 34.061910492999999, 2024-04-19T02:04:04.351 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.025249772, 2024-04-19T02:04:04.351 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:04.351 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:04.351 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:04.351 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:04.351 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.351 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:04.351 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.209587+0000", 2024-04-19T02:04:04.351 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.351 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.351 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.351 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:04.352 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.209587+0000", 2024-04-19T02:04:04.352 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.352 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.352 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.352 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:04.352 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.209588+0000", 2024-04-19T02:04:04.352 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.4099999999999999e-07 2024-04-19T02:04:04.352 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.352 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.352 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:04.352 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.209834+0000", 2024-04-19T02:04:04.352 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024650200000000001 2024-04-19T02:04:04.352 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.352 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.353 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:04.353 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.209854+0000", 2024-04-19T02:04:04.353 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9758999999999998e-05 2024-04-19T02:04:04.353 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.353 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.353 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.353 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.209867+0000", 2024-04-19T02:04:04.353 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2917e-05 2024-04-19T02:04:04.353 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.353 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.353 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:04.353 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.234535+0000", 2024-04-19T02:04:04.353 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.024668606999999999 2024-04-19T02:04:04.354 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.354 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.354 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:04.354 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.234837+0000", 2024-04-19T02:04:04.354 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00030144599999999997 2024-04-19T02:04:04.354 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.354 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:04.354 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.354 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.354 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.354 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 46/26 ECSubWriteReply(tid=164, last_complete=46'79, committed=1, applied=1))", 2024-04-19T02:04:04.354 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.210983+0000", 2024-04-19T02:04:04.354 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 34.060514685999998, 2024-04-19T02:04:04.355 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.024646257000000001, 2024-04-19T02:04:04.355 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:04.355 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:04.355 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:04.355 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:04.355 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.355 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:04.355 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210983+0000", 2024-04-19T02:04:04.355 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.355 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.355 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.355 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:04.355 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210983+0000", 2024-04-19T02:04:04.355 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.355 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.356 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.356 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:04.356 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210983+0000", 2024-04-19T02:04:04.356 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2100000000000002e-07 2024-04-19T02:04:04.356 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.356 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.356 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:04.356 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210987+0000", 2024-04-19T02:04:04.356 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2789999999999999e-06 2024-04-19T02:04:04.356 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.356 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.356 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:04.356 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210995+0000", 2024-04-19T02:04:04.356 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3790000000000006e-06 2024-04-19T02:04:04.357 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.357 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.357 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.357 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.211007+0000", 2024-04-19T02:04:04.357 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2373e-05 2024-04-19T02:04:04.357 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.357 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.357 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:04.357 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235615+0000", 2024-04-19T02:04:04.357 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.024607688999999999 2024-04-19T02:04:04.357 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.357 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.357 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:04.357 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235629+0000", 2024-04-19T02:04:04.358 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4116e-05 2024-04-19T02:04:04.358 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.358 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:04.358 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.358 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.358 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.358 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 46/26 ECSubWriteReply(tid=167, last_complete=46'81, committed=1, applied=1))", 2024-04-19T02:04:04.358 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.211022+0000", 2024-04-19T02:04:04.358 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 34.060475801000003, 2024-04-19T02:04:04.358 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.024969574000000001, 2024-04-19T02:04:04.358 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:04.358 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:04.358 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:04.359 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:04.359 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.359 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:04.359 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.211022+0000", 2024-04-19T02:04:04.359 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.359 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.359 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.359 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:04.359 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.211022+0000", 2024-04-19T02:04:04.359 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.359 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.359 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.359 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:04.359 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.211022+0000", 2024-04-19T02:04:04.359 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2000000000000001e-07 2024-04-19T02:04:04.360 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.360 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.360 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:04.360 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.211025+0000", 2024-04-19T02:04:04.360 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.717e-06 2024-04-19T02:04:04.360 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.360 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.360 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:04.360 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.218661+0000", 2024-04-19T02:04:04.360 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0076366139999999999 2024-04-19T02:04:04.360 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.360 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.360 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.361 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.218669+0000", 2024-04-19T02:04:04.361 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.678e-06 2024-04-19T02:04:04.361 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.361 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.361 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:04.361 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235978+0000", 2024-04-19T02:04:04.361 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.017308430999999999 2024-04-19T02:04:04.361 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.361 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.361 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:04.361 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235991+0000", 2024-04-19T02:04:04.361 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3814e-05 2024-04-19T02:04:04.361 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.361 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:04.362 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.362 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.362 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.362 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:888 3.2s0 3:465821b6:::smithi05930959-32:head [rollback c] snapc 1d=[1d,1c,1b,19,17,16,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-04-19T02:04:04.362 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:04:00.154360+0000", 2024-04-19T02:04:04.362 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 4.1171379689999998, 2024-04-19T02:04:04.362 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.027291403999999998, 2024-04-19T02:04:04.362 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:04.362 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:04.362 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:04.362 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:04.362 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:04.362 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.362 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 888 2024-04-19T02:04:04.363 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.363 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:04.363 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.363 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:04.363 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.154360+0000", 2024-04-19T02:04:04.363 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.363 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.363 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.363 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:04.363 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.154360+0000", 2024-04-19T02:04:04.363 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.363 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.363 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.363 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:04.364 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.154360+0000", 2024-04-19T02:04:04.364 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0100000000000004e-07 2024-04-19T02:04:04.364 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.364 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.364 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:04.364 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.154365+0000", 2024-04-19T02:04:04.364 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1330000000000001e-06 2024-04-19T02:04:04.364 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.364 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.364 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:04.364 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.154375+0000", 2024-04-19T02:04:04.364 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0652000000000001e-05 2024-04-19T02:04:04.364 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.364 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.365 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.365 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.154383+0000", 2024-04-19T02:04:04.365 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.1859999999999992e-06 2024-04-19T02:04:04.365 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.365 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.365 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:04.365 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.157163+0000", 2024-04-19T02:04:04.365 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0027800260000000001 2024-04-19T02:04:04.365 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.365 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.365 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:04.365 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.157263+0000", 2024-04-19T02:04:04.365 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9890999999999996e-05 2024-04-19T02:04:04.365 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.365 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.366 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:04.366 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.157926+0000", 2024-04-19T02:04:04.366 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00066291400000000002 2024-04-19T02:04:04.366 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.366 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.366 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:04.366 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.160288+0000", 2024-04-19T02:04:04.366 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0023615440000000001 2024-04-19T02:04:04.366 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.366 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.366 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:04.366 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.181562+0000", 2024-04-19T02:04:04.366 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.021273888000000001 2024-04-19T02:04:04.367 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.367 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.367 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:04.367 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.181651+0000", 2024-04-19T02:04:04.367 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.9369000000000004e-05 2024-04-19T02:04:04.367 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.367 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:04.367 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.367 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.367 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.367 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:889 3.2s0 3:465821b6:::smithi05930959-32:head [append 0~0] snapc 1d=[1d,1c,1b,19,17,16,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-04-19T02:04:04.367 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:04:00.154397+0000", 2024-04-19T02:04:04.367 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 4.1171003769999999, 2024-04-19T02:04:04.367 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.027438435000000001, 2024-04-19T02:04:04.368 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:04.368 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:04.368 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:04.368 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:04.368 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:04.368 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.368 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 889 2024-04-19T02:04:04.368 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.368 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:04.368 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.368 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:04.368 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.154397+0000", 2024-04-19T02:04:04.368 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.368 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.369 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.369 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:04.369 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.154397+0000", 2024-04-19T02:04:04.369 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:04.369 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.369 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.369 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:04.369 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.154398+0000", 2024-04-19T02:04:04.369 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9400000000000005e-07 2024-04-19T02:04:04.369 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.369 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.369 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:04.369 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.154402+0000", 2024-04-19T02:04:04.369 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4240000000000003e-06 2024-04-19T02:04:04.370 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.370 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.370 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:04.370 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.154414+0000", 2024-04-19T02:04:04.370 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1705999999999999e-05 2024-04-19T02:04:04.370 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.370 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.370 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.370 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.154422+0000", 2024-04-19T02:04:04.370 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2369999999999992e-06 2024-04-19T02:04:04.370 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.370 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.370 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:04.370 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.158437+0000", 2024-04-19T02:04:04.371 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0040149929999999997 2024-04-19T02:04:04.371 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.371 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.371 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:04.371 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.158567+0000", 2024-04-19T02:04:04.371 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000130025 2024-04-19T02:04:04.371 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.371 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.371 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:04.371 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.159870+0000", 2024-04-19T02:04:04.371 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0013025949999999999 2024-04-19T02:04:04.371 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.371 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.371 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:04.372 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.162624+0000", 2024-04-19T02:04:04.372 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002754126 2024-04-19T02:04:04.372 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.372 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.372 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:04.372 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.181770+0000", 2024-04-19T02:04:04.372 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.019145549000000001 2024-04-19T02:04:04.372 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:04.372 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:04.372 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:04.372 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.181836+0000", 2024-04-19T02:04:04.372 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.6086000000000004e-05 2024-04-19T02:04:04.372 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.372 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:04.372 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.373 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:04.373 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:04.373 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:04:04.374 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-19T02:04:04.375 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:04:04.373+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:04:04.475 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:04:04.473+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:04:04.504 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:04:04.505 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:04:04.505 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 0 2024-04-19T02:04:04.505 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:04:04.519 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-19T02:04:04.576 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:04:04.577+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:04:04.643 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:04:04.643 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:04:04.643 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-04-19T02:04:04.643 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-04-19T02:04:04.643 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:04:04.658 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-19T02:04:04.677 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:04:04.677+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:04:04.778 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:04:04.781+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:04:04.785 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:04:04.786 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-04-19T02:04:04.786 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-04-19T02:04:04.786 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-04-19T02:04:04.786 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.786 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.cs4 46/26 ECSubWrite(tid=44, reqid=osd.9.0:44, at_version=46'26, trim_to=0'0, roll_forward_to=45'24))", 2024-04-19T02:04:04.786 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.065744+0000", 2024-04-19T02:04:04.786 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 34.719515082999997, 2024-04-19T02:04:04.786 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.018889659, 2024-04-19T02:04:04.786 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:04.786 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:04.786 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:04.787 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:04.787 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.787 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:04.787 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.065744+0000", 2024-04-19T02:04:04.787 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.787 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.787 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.787 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:04.787 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.065744+0000", 2024-04-19T02:04:04.787 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.787 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.787 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.788 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:04.788 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.065745+0000", 2024-04-19T02:04:04.788 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0999999999999998e-07 2024-04-19T02:04:04.788 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.788 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.788 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:04.788 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.065794+0000", 2024-04-19T02:04:04.788 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9177e-05 2024-04-19T02:04:04.788 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.788 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.788 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:04.788 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.065836+0000", 2024-04-19T02:04:04.789 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2245e-05 2024-04-19T02:04:04.789 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.789 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.789 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.789 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.065860+0000", 2024-04-19T02:04:04.789 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3654000000000001e-05 2024-04-19T02:04:04.789 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.789 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.789 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:04.789 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066138+0000", 2024-04-19T02:04:04.789 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00027793700000000002 2024-04-19T02:04:04.789 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.789 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.790 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:04.790 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.066164+0000", 2024-04-19T02:04:04.790 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5862000000000001e-05 2024-04-19T02:04:04.790 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.790 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.790 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:04.790 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.084533+0000", 2024-04-19T02:04:04.790 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.018369534 2024-04-19T02:04:04.790 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.790 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.790 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:04.790 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.084634+0000", 2024-04-19T02:04:04.791 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010054 2024-04-19T02:04:04.791 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.791 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:04.791 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.791 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.791 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.791 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:432 3.8s0 3:1984c4bc:::smithi05930959-40:head [read 0~3162112,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:04.791 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.193759+0000", 2024-04-19T02:04:04.791 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 34.591499900000002, 2024-04-19T02:04:04.791 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.020892661, 2024-04-19T02:04:04.791 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:04.791 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:04.791 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:04:04.792 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:04:04.792 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:04:04.792 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.792 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 432 2024-04-19T02:04:04.792 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.792 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:04.792 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.792 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:04.792 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193759+0000", 2024-04-19T02:04:04.792 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.792 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.792 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.793 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:04.793 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193759+0000", 2024-04-19T02:04:04.793 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.793 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.793 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.793 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:04.793 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193761+0000", 2024-04-19T02:04:04.793 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3430000000000001e-06 2024-04-19T02:04:04.793 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.793 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.793 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:04.793 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193770+0000", 2024-04-19T02:04:04.794 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5359999999999998e-06 2024-04-19T02:04:04.794 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.794 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.794 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:04.794 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193794+0000", 2024-04-19T02:04:04.794 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3589000000000001e-05 2024-04-19T02:04:04.794 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.794 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.794 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.794 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193811+0000", 2024-04-19T02:04:04.794 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6838999999999999e-05 2024-04-19T02:04:04.794 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.795 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.795 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:04.795 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.194657+0000", 2024-04-19T02:04:04.795 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00084614800000000002 2024-04-19T02:04:04.795 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.795 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.795 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:04:04.795 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.194925+0000", 2024-04-19T02:04:04.795 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000267927 2024-04-19T02:04:04.795 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.795 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.795 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:04.796 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.214652+0000", 2024-04-19T02:04:04.796 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.019727279 2024-04-19T02:04:04.796 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.796 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:04.796 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.796 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.796 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.796 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:433 3.8s0 3:1984c4bc:::smithi05930959-40:head [read 0~3162112,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:04.796 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.193839+0000", 2024-04-19T02:04:04.796 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 34.591420481999997, 2024-04-19T02:04:04.796 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.025594295, 2024-04-19T02:04:04.796 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:04.796 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:04.797 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:04:04.797 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:04:04.797 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:04:04.797 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.797 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 433 2024-04-19T02:04:04.797 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.797 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:04.797 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.797 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:04.797 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193839+0000", 2024-04-19T02:04:04.797 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.797 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.798 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.798 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:04.798 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193839+0000", 2024-04-19T02:04:04.798 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.798 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.798 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.798 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:04.798 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193840+0000", 2024-04-19T02:04:04.798 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.277e-06 2024-04-19T02:04:04.798 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.798 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.798 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:04.799 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193848+0000", 2024-04-19T02:04:04.799 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0790000000000001e-06 2024-04-19T02:04:04.799 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.799 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.799 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:04.799 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193866+0000", 2024-04-19T02:04:04.799 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8411999999999999e-05 2024-04-19T02:04:04.799 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.799 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.799 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.799 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.193881+0000", 2024-04-19T02:04:04.799 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4419e-05 2024-04-19T02:04:04.800 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.800 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.800 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:04.800 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.202091+0000", 2024-04-19T02:04:04.800 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0082103079999999995 2024-04-19T02:04:04.800 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.800 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.800 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:04:04.800 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.202540+0000", 2024-04-19T02:04:04.800 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00044865600000000002 2024-04-19T02:04:04.800 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.800 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.800 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:04.801 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.219433+0000", 2024-04-19T02:04:04.801 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.016893143999999999 2024-04-19T02:04:04.801 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.801 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:04.801 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.801 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.801 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.801 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:576 3.8s0 3:16258ed3:::smithi05930959-3:head [append 0~655360] snapc 10=[10,f,c] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-19T02:04:04.801 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:40.161424+0000", 2024-04-19T02:04:04.801 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 24.623834884000001, 2024-04-19T02:04:04.801 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.018735465, 2024-04-19T02:04:04.801 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:04.801 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:04.802 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:04.802 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:04:04.802 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:04:04.802 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.802 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 576 2024-04-19T02:04:04.802 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.802 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:04.803 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.803 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:04.803 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.161424+0000", 2024-04-19T02:04:04.803 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.803 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.803 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.803 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:04.803 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.161424+0000", 2024-04-19T02:04:04.803 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.803 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.804 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.804 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:04.804 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.161428+0000", 2024-04-19T02:04:04.804 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.8670000000000004e-06 2024-04-19T02:04:04.804 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.804 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.804 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:04.804 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.162988+0000", 2024-04-19T02:04:04.804 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001559476 2024-04-19T02:04:04.804 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.804 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.804 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:04.804 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.163028+0000", 2024-04-19T02:04:04.805 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.9950999999999997e-05 2024-04-19T02:04:04.805 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.805 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.805 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.805 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.163047+0000", 2024-04-19T02:04:04.805 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9162000000000001e-05 2024-04-19T02:04:04.805 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.805 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.805 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:04.805 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.163203+0000", 2024-04-19T02:04:04.805 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015594 2024-04-19T02:04:04.805 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.806 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.806 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "op must wait for map", 2024-04-19T02:04:04.806 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.163215+0000", 2024-04-19T02:04:04.806 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2748999999999999e-05 2024-04-19T02:04:04.806 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.806 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.806 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:04.806 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.166369+0000", 2024-04-19T02:04:04.806 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0031538080000000001 2024-04-19T02:04:04.806 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.806 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.806 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:04:04.807 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.166492+0000", 2024-04-19T02:04:04.807 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012248900000000001 2024-04-19T02:04:04.807 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.807 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.807 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:04.807 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.168148+0000", 2024-04-19T02:04:04.807 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0016566809999999999 2024-04-19T02:04:04.807 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.807 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.807 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:04.807 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.170497+0000", 2024-04-19T02:04:04.807 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0023481830000000002 2024-04-19T02:04:04.807 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.808 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.808 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:04:04.808 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.180068+0000", 2024-04-19T02:04:04.808 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0095713729999999993 2024-04-19T02:04:04.808 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.808 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.808 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:04.808 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:40.180160+0000", 2024-04-19T02:04:04.808 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1786000000000005e-05 2024-04-19T02:04:04.808 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.808 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:04.808 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.808 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.809 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.809 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:722 3.8s0 3:1984c4bc:::smithi05930959-40:head [append 0~0] snapc 16=[16,15,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-19T02:04:04.809 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:50.059058+0000", 2024-04-19T02:04:04.809 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 14.726201566, 2024-04-19T02:04:04.809 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.024483515000000001, 2024-04-19T02:04:04.809 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:04.809 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:04.809 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:04.809 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:04:04.809 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:04:04.809 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.809 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 722 2024-04-19T02:04:04.809 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.810 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:04.810 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.810 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:04.810 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059058+0000", 2024-04-19T02:04:04.810 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.810 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.810 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.810 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:04.810 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059058+0000", 2024-04-19T02:04:04.810 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.810 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.810 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.810 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:04.811 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059060+0000", 2024-04-19T02:04:04.811 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.807e-06 2024-04-19T02:04:04.811 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.811 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.811 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:04.811 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059074+0000", 2024-04-19T02:04:04.811 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3746e-05 2024-04-19T02:04:04.811 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.811 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.811 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:04.811 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059107+0000", 2024-04-19T02:04:04.811 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3222000000000001e-05 2024-04-19T02:04:04.812 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.812 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.812 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.812 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059125+0000", 2024-04-19T02:04:04.812 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7176e-05 2024-04-19T02:04:04.812 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.812 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.812 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:04.812 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059230+0000", 2024-04-19T02:04:04.812 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000105707 2024-04-19T02:04:04.812 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.812 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.812 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:04:04.813 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059396+0000", 2024-04-19T02:04:04.813 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016555000000000001 2024-04-19T02:04:04.813 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.813 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.813 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:04.813 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.061077+0000", 2024-04-19T02:04:04.813 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0016816279999999999 2024-04-19T02:04:04.813 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.813 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.813 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:04.813 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.072604+0000", 2024-04-19T02:04:04.813 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011526773000000001 2024-04-19T02:04:04.813 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.814 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.814 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:04:04.814 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.083326+0000", 2024-04-19T02:04:04.814 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010722206999999999 2024-04-19T02:04:04.814 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.814 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.814 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:04.814 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.083541+0000", 2024-04-19T02:04:04.814 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000214699 2024-04-19T02:04:04.814 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.814 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:04.814 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.815 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.815 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.815 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:723 3.8s0 3:1984c4bc:::smithi05930959-40:head [rollback 10] snapc 16=[16,15,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-19T02:04:04.815 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:50.059158+0000", 2024-04-19T02:04:04.815 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 14.726101289000001, 2024-04-19T02:04:04.815 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.024958859999999999, 2024-04-19T02:04:04.815 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:04.815 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:04.815 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:04.815 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:04:04.815 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:04:04.815 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.816 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 723 2024-04-19T02:04:04.816 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.816 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:04.816 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.816 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:04.816 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059158+0000", 2024-04-19T02:04:04.816 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.816 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.816 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.816 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:04.816 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059158+0000", 2024-04-19T02:04:04.816 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.816 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.817 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.817 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:04.817 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059159+0000", 2024-04-19T02:04:04.817 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.299e-06 2024-04-19T02:04:04.817 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.817 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.817 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:04.817 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059166+0000", 2024-04-19T02:04:04.817 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4640000000000003e-06 2024-04-19T02:04:04.817 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.817 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.817 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:04.817 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059186+0000", 2024-04-19T02:04:04.818 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0058999999999999e-05 2024-04-19T02:04:04.818 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.818 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.818 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.818 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059201+0000", 2024-04-19T02:04:04.818 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5775999999999999e-05 2024-04-19T02:04:04.818 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.818 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.818 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:04.818 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.062943+0000", 2024-04-19T02:04:04.818 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0037411559999999998 2024-04-19T02:04:04.818 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.819 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.819 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:04:04.819 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.063225+0000", 2024-04-19T02:04:04.819 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00028223300000000001 2024-04-19T02:04:04.819 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.819 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.819 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:04.819 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.064807+0000", 2024-04-19T02:04:04.819 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001582518 2024-04-19T02:04:04.819 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.819 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.819 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:04.820 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.072608+0000", 2024-04-19T02:04:04.820 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0078006339999999999 2024-04-19T02:04:04.820 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.820 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.820 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:04:04.820 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.083936+0000", 2024-04-19T02:04:04.820 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011328474 2024-04-19T02:04:04.820 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.820 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.820 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:04.820 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.084117+0000", 2024-04-19T02:04:04.820 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000180247 2024-04-19T02:04:04.820 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.821 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:04.821 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.821 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.821 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.821 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:724 3.8s0 3:1984c4bc:::smithi05930959-40:head [append 0~0] snapc 16=[16,15,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-19T02:04:04.821 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:50.059247+0000", 2024-04-19T02:04:04.821 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 14.726012365999999, 2024-04-19T02:04:04.821 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.031205883, 2024-04-19T02:04:04.821 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:04.822 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:04.822 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:04.822 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:04:04.822 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:04:04.822 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.822 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 724 2024-04-19T02:04:04.822 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.822 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:04.822 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.822 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:04.822 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059247+0000", 2024-04-19T02:04:04.822 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.823 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.823 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.823 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:04.823 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059247+0000", 2024-04-19T02:04:04.823 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.823 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.823 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.823 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:04.823 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059249+0000", 2024-04-19T02:04:04.823 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.807e-06 2024-04-19T02:04:04.823 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.823 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.824 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:04.824 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059255+0000", 2024-04-19T02:04:04.824 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6660000000000002e-06 2024-04-19T02:04:04.824 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.824 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.824 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:04.824 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059273+0000", 2024-04-19T02:04:04.824 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8145999999999999e-05 2024-04-19T02:04:04.824 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.824 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.825 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.825 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059286+0000", 2024-04-19T02:04:04.825 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2792999999999999e-05 2024-04-19T02:04:04.825 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.825 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.825 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:04.825 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.066337+0000", 2024-04-19T02:04:04.825 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0070504840000000001 2024-04-19T02:04:04.825 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.825 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.825 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:04:04.825 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.066602+0000", 2024-04-19T02:04:04.825 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00026546399999999998 2024-04-19T02:04:04.826 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.826 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.826 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:04.826 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.068444+0000", 2024-04-19T02:04:04.826 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0018421819999999999 2024-04-19T02:04:04.826 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.826 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.826 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:04.826 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.072610+0000", 2024-04-19T02:04:04.826 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0041654200000000004 2024-04-19T02:04:04.826 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.826 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.827 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:04:04.827 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.090353+0000", 2024-04-19T02:04:04.827 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.017743024999999999 2024-04-19T02:04:04.827 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.827 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.827 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:04.827 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.090453+0000", 2024-04-19T02:04:04.827 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9895999999999994e-05 2024-04-19T02:04:04.827 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.827 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:04.827 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.827 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.828 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.828 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:725 3.8s0 3:16258ed3:::smithi05930959-3:16 [read 0~3620864,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-04-19T02:04:04.828 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:50.059299+0000", 2024-04-19T02:04:04.828 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 14.725959741, 2024-04-19T02:04:04.828 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.030321817000000001, 2024-04-19T02:04:04.828 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:04.828 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:04.828 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:04:04.828 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:04:04.828 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:04:04.828 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.828 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 725 2024-04-19T02:04:04.829 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.829 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:04.829 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.829 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:04.829 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059299+0000", 2024-04-19T02:04:04.829 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.829 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.829 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.829 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:04.829 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059299+0000", 2024-04-19T02:04:04.829 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.829 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.830 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.830 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:04.830 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059300+0000", 2024-04-19T02:04:04.830 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0330000000000001e-06 2024-04-19T02:04:04.830 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.830 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.830 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:04.830 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059306+0000", 2024-04-19T02:04:04.830 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0000000000000002e-06 2024-04-19T02:04:04.830 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.830 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.830 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:04.830 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059317+0000", 2024-04-19T02:04:04.831 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0813e-05 2024-04-19T02:04:04.831 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.831 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.831 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.831 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059325+0000", 2024-04-19T02:04:04.831 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0989999999999999e-06 2024-04-19T02:04:04.831 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.831 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.831 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:04.831 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.069664+0000", 2024-04-19T02:04:04.831 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010338257 2024-04-19T02:04:04.831 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.831 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.832 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:04:04.832 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.069951+0000", 2024-04-19T02:04:04.832 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00028775300000000002 2024-04-19T02:04:04.832 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.832 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.832 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:04.832 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.089621+0000", 2024-04-19T02:04:04.832 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.019669862 2024-04-19T02:04:04.832 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.832 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:04.832 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.832 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.833 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.833 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:726 3.8s0 3:16258ed3:::smithi05930959-3:16 [read 0~3620864,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-04-19T02:04:04.833 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:50.059335+0000", 2024-04-19T02:04:04.833 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 14.725923816, 2024-04-19T02:04:04.833 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.037495316000000001, 2024-04-19T02:04:04.833 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:04.833 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:04.833 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:04:04.833 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:04:04.833 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:04:04.833 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.833 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 726 2024-04-19T02:04:04.833 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.834 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:04.834 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.834 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:04.834 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059335+0000", 2024-04-19T02:04:04.834 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.834 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.834 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.834 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:04.834 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059335+0000", 2024-04-19T02:04:04.834 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.834 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.834 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.835 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:04.835 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059336+0000", 2024-04-19T02:04:04.835 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6700000000000003e-07 2024-04-19T02:04:04.835 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.835 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.835 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:04.835 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059340+0000", 2024-04-19T02:04:04.835 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.3170000000000003e-06 2024-04-19T02:04:04.836 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.836 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.836 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:04.836 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059354+0000", 2024-04-19T02:04:04.836 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3925e-05 2024-04-19T02:04:04.836 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.836 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.836 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.836 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059363+0000", 2024-04-19T02:04:04.836 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.1800000000000002e-06 2024-04-19T02:04:04.836 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.836 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.837 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:04.837 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.070839+0000", 2024-04-19T02:04:04.837 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011475992000000001 2024-04-19T02:04:04.837 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.837 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.837 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:04:04.837 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.071105+0000", 2024-04-19T02:04:04.837 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00026514000000000003 2024-04-19T02:04:04.837 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.837 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.837 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:04.838 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.096831+0000", 2024-04-19T02:04:04.838 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.025726095000000001 2024-04-19T02:04:04.838 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.838 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:04.838 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.838 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.838 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.838 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:727 3.8s0 3:16258ed3:::smithi05930959-3:16 [read 0~3620864,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-04-19T02:04:04.838 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:50.059373+0000", 2024-04-19T02:04:04.838 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 14.72588659, 2024-04-19T02:04:04.838 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.046336992, 2024-04-19T02:04:04.838 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:04.839 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:04.839 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:04:04.839 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:04:04.839 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:04:04.839 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.839 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 727 2024-04-19T02:04:04.839 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.839 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:04.839 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.839 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:04.839 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059373+0000", 2024-04-19T02:04:04.840 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.840 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.840 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.840 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:04.840 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059373+0000", 2024-04-19T02:04:04.840 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.840 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.840 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.840 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:04.840 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059373+0000", 2024-04-19T02:04:04.840 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.3900000000000004e-07 2024-04-19T02:04:04.840 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.840 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.841 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:04.841 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059377+0000", 2024-04-19T02:04:04.841 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.0099999999999997e-06 2024-04-19T02:04:04.841 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.841 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.841 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:04.841 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059387+0000", 2024-04-19T02:04:04.842 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.5230000000000001e-06 2024-04-19T02:04:04.842 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.842 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.842 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.842 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.059394+0000", 2024-04-19T02:04:04.842 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.374e-06 2024-04-19T02:04:04.842 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.842 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.842 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:04.842 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.096990+0000", 2024-04-19T02:04:04.842 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.037595295000000001 2024-04-19T02:04:04.842 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.843 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.843 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:04:04.843 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.097136+0000", 2024-04-19T02:04:04.843 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000146162 2024-04-19T02:04:04.843 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.843 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.843 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:04.843 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.105710+0000", 2024-04-19T02:04:04.843 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0085738889999999995 2024-04-19T02:04:04.843 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.843 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:04.843 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.843 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.844 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.844 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:731 3.9s0 3:97177082:::smithi05930959-45:12 [read 0~1572864,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-04-19T02:04:04.844 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:50.074982+0000", 2024-04-19T02:04:04.844 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 14.710277584, 2024-04-19T02:04:04.844 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.02096114, 2024-04-19T02:04:04.844 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:04.844 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:04.844 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:04:04.844 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:04:04.844 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:04:04.844 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.845 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 731 2024-04-19T02:04:04.845 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.845 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:04.845 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.845 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:04.845 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.074982+0000", 2024-04-19T02:04:04.845 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.845 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.845 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.845 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:04.845 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.074982+0000", 2024-04-19T02:04:04.845 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.845 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.846 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.846 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:04.846 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.074983+0000", 2024-04-19T02:04:04.846 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1570000000000001e-06 2024-04-19T02:04:04.846 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.846 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.846 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:04.846 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.074993+0000", 2024-04-19T02:04:04.846 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0033e-05 2024-04-19T02:04:04.846 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.846 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.846 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:04.847 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.075011+0000", 2024-04-19T02:04:04.847 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8601999999999999e-05 2024-04-19T02:04:04.847 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.847 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.847 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.847 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.075032+0000", 2024-04-19T02:04:04.847 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0939999999999999e-05 2024-04-19T02:04:04.847 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.847 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.847 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:04.847 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.085459+0000", 2024-04-19T02:04:04.847 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010426327000000001 2024-04-19T02:04:04.847 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.848 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.848 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:04:04.848 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.085750+0000", 2024-04-19T02:04:04.848 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00029137599999999998 2024-04-19T02:04:04.848 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.848 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.848 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:04.848 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.095943+0000", 2024-04-19T02:04:04.848 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010192705 2024-04-19T02:04:04.848 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.848 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:04.848 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.848 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.849 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.849 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpReadReply(3.8s0 66/26 ECSubReadReply(tid=154, attrs_read=0))", 2024-04-19T02:04:04.849 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:50.077044+0000", 2024-04-19T02:04:04.849 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 14.708214666, 2024-04-19T02:04:04.849 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.019872520000000001, 2024-04-19T02:04:04.849 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:04.849 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:04.849 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:04.849 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:04.849 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.849 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:04.849 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.077044+0000", 2024-04-19T02:04:04.850 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.850 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.850 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.850 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:04.850 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.077044+0000", 2024-04-19T02:04:04.850 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.850 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.850 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.850 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:04.850 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.077046+0000", 2024-04-19T02:04:04.850 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.19e-06 2024-04-19T02:04:04.850 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.851 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.851 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:04.851 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.078710+0000", 2024-04-19T02:04:04.851 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001664355 2024-04-19T02:04:04.851 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.851 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.851 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:04.851 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.078739+0000", 2024-04-19T02:04:04.851 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8849000000000001e-05 2024-04-19T02:04:04.851 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.851 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.851 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.852 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.078756+0000", 2024-04-19T02:04:04.852 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7382999999999999e-05 2024-04-19T02:04:04.852 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.852 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.852 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:04.852 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.092525+0000", 2024-04-19T02:04:04.852 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013768591 2024-04-19T02:04:04.852 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.852 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.852 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:04.852 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.096917+0000", 2024-04-19T02:04:04.852 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0043921519999999999 2024-04-19T02:04:04.853 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.853 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:04.853 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.853 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.853 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.853 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:755 3.8s0 3:1c5eeecd:::smithi05930959-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e66)", 2024-04-19T02:04:04.853 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:50.389519+0000", 2024-04-19T02:04:04.853 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 14.395740189, 2024-04-19T02:04:04.853 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.020545033000000001, 2024-04-19T02:04:04.853 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:04.853 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:04.853 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:04:04.853 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:04:04.854 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:04:04.854 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.854 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 755 2024-04-19T02:04:04.854 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.854 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:04.854 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.854 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:04.854 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.389519+0000", 2024-04-19T02:04:04.854 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.854 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.854 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.854 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:04.855 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.389519+0000", 2024-04-19T02:04:04.855 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.855 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.855 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.855 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:04.855 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.389520+0000", 2024-04-19T02:04:04.855 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6199999999999998e-07 2024-04-19T02:04:04.855 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.855 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.855 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:04.855 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.389522+0000", 2024-04-19T02:04:04.855 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5349999999999999e-06 2024-04-19T02:04:04.856 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.856 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.856 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:04.856 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.389530+0000", 2024-04-19T02:04:04.856 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7600000000000002e-06 2024-04-19T02:04:04.856 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.856 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.856 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.856 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.389536+0000", 2024-04-19T02:04:04.856 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6879999999999996e-06 2024-04-19T02:04:04.856 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.856 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.856 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:04.856 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.403906+0000", 2024-04-19T02:04:04.857 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014370106000000001 2024-04-19T02:04:04.857 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.857 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.857 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for rw locks", 2024-04-19T02:04:04.857 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.404154+0000", 2024-04-19T02:04:04.857 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000248154 2024-04-19T02:04:04.857 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.857 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.857 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:04.857 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.406109+0000", 2024-04-19T02:04:04.857 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0019556780000000002 2024-04-19T02:04:04.857 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.857 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.858 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:04:04.858 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.406192+0000", 2024-04-19T02:04:04.858 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3040000000000005e-05 2024-04-19T02:04:04.858 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.858 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.858 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:04.858 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.410064+0000", 2024-04-19T02:04:04.858 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0038715099999999999 2024-04-19T02:04:04.858 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.858 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:04.858 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.858 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.859 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.859 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:870 3.8s0 3:1984c4bc:::smithi05930959-40:head [copy-from ver 141] snapc 1d=[1d,1c,1b,19,17,16,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-04-19T02:04:04.859 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:04:00.123980+0000", 2024-04-19T02:04:04.859 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.6612791380000003, 2024-04-19T02:04:04.859 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.059512838999999998, 2024-04-19T02:04:04.859 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:04.859 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:04.859 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:04.859 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:04:04.860 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:04:04.860 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.860 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 870 2024-04-19T02:04:04.860 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.860 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:04.860 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.860 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:04.860 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.123980+0000", 2024-04-19T02:04:04.860 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.860 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.860 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.860 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:04.860 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.123980+0000", 2024-04-19T02:04:04.861 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.861 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.861 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.861 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:04.861 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.123982+0000", 2024-04-19T02:04:04.861 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.717e-06 2024-04-19T02:04:04.861 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.861 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.861 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:04.861 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.123991+0000", 2024-04-19T02:04:04.861 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.3209999999999994e-06 2024-04-19T02:04:04.861 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.862 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.862 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:04.862 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.124005+0000", 2024-04-19T02:04:04.862 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4054e-05 2024-04-19T02:04:04.862 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.862 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.862 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.862 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.124013+0000", 2024-04-19T02:04:04.862 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.1370000000000002e-06 2024-04-19T02:04:04.862 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.862 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.862 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:04.863 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.124097+0000", 2024-04-19T02:04:04.863 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.3337000000000005e-05 2024-04-19T02:04:04.863 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.863 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.863 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:04:04.863 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.124255+0000", 2024-04-19T02:04:04.863 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00015835 2024-04-19T02:04:04.863 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.863 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.863 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:04.863 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.159408+0000", 2024-04-19T02:04:04.863 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.035153007 2024-04-19T02:04:04.863 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.864 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.864 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:04.864 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.169898+0000", 2024-04-19T02:04:04.864 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010490506 2024-04-19T02:04:04.864 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.864 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.864 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "commit_sent", 2024-04-19T02:04:04.864 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.183382+0000", 2024-04-19T02:04:04.864 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013483451 2024-04-19T02:04:04.864 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.864 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.864 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:04.864 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.183493+0000", 2024-04-19T02:04:04.865 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00011095900000000001 2024-04-19T02:04:04.865 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.865 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:04.865 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.865 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.865 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.865 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:871 3.8s0 3:1984c4bc:::smithi05930959-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e76)", 2024-04-19T02:04:04.865 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:04:00.124032+0000", 2024-04-19T02:04:04.865 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.6612269699999995, 2024-04-19T02:04:04.865 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.041729954999999999, 2024-04-19T02:04:04.865 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:04.865 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:04.865 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:04:04.866 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:04:04.866 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:04:04.866 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.866 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 871 2024-04-19T02:04:04.866 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.866 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:04.866 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.866 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:04.866 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.124032+0000", 2024-04-19T02:04:04.866 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.866 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.866 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.866 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:04.867 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.124032+0000", 2024-04-19T02:04:04.867 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.867 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.867 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.867 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:04.867 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.124033+0000", 2024-04-19T02:04:04.867 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7299999999999995e-07 2024-04-19T02:04:04.867 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.867 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.867 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:04.867 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.124036+0000", 2024-04-19T02:04:04.867 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.971e-06 2024-04-19T02:04:04.868 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.868 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.868 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:04.868 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.124044+0000", 2024-04-19T02:04:04.868 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.6559999999999998e-06 2024-04-19T02:04:04.868 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.868 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.868 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.868 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.124052+0000", 2024-04-19T02:04:04.868 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.0039999999999999e-06 2024-04-19T02:04:04.868 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.868 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.868 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:04.869 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.124626+0000", 2024-04-19T02:04:04.869 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00057304800000000005 2024-04-19T02:04:04.869 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.869 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.869 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "waiting for blocked object", 2024-04-19T02:04:04.869 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.124713+0000", 2024-04-19T02:04:04.869 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7325999999999997e-05 2024-04-19T02:04:04.869 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.869 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.869 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:04.869 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.165471+0000", 2024-04-19T02:04:04.869 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.040757847 2024-04-19T02:04:04.869 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.870 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.870 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:04:04.870 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.165612+0000", 2024-04-19T02:04:04.870 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00014129499999999999 2024-04-19T02:04:04.870 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.870 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.870 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:04.870 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.165762+0000", 2024-04-19T02:04:04.870 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000150135 2024-04-19T02:04:04.870 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.870 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:04.870 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.870 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.871 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.871 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:878 3.9s0 3:956ba687:::smithi05930959-13:head [read 0~3342336,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-04-19T02:04:04.871 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:04:00.149098+0000", 2024-04-19T02:04:04.871 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.6361608929999996, 2024-04-19T02:04:04.871 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.024460972000000001, 2024-04-19T02:04:04.871 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:04.871 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:04.871 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:04:04.871 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:04:04.871 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:04:04.871 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.871 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 878 2024-04-19T02:04:04.872 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.872 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:04.872 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.872 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:04.872 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.149098+0000", 2024-04-19T02:04:04.872 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.872 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.872 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.872 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:04.872 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.149098+0000", 2024-04-19T02:04:04.872 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.872 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.872 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.873 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:04.873 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.149101+0000", 2024-04-19T02:04:04.873 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8159999999999998e-06 2024-04-19T02:04:04.873 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.873 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.873 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:04.873 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.149115+0000", 2024-04-19T02:04:04.873 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3706999999999999e-05 2024-04-19T02:04:04.873 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.873 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.873 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:04.873 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.149148+0000", 2024-04-19T02:04:04.873 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3151000000000002e-05 2024-04-19T02:04:04.874 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.874 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.874 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.874 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.149164+0000", 2024-04-19T02:04:04.874 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.5705999999999998e-05 2024-04-19T02:04:04.874 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.874 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.874 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:04.874 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.149271+0000", 2024-04-19T02:04:04.874 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010723 2024-04-19T02:04:04.874 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.874 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.874 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:04:04.875 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.149442+0000", 2024-04-19T02:04:04.875 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017144 2024-04-19T02:04:04.875 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.875 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.875 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:04.875 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.173559+0000", 2024-04-19T02:04:04.875 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.024116921999999999 2024-04-19T02:04:04.875 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.875 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:04.875 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.875 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.876 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.876 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:879 3.9s0 3:956ba687:::smithi05930959-13:head [read 0~3342336,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-04-19T02:04:04.876 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:04:00.149201+0000", 2024-04-19T02:04:04.876 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.6360581009999997, 2024-04-19T02:04:04.876 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.028274193999999999, 2024-04-19T02:04:04.876 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:04.876 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:04.876 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:04:04.876 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:04:04.876 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:04:04.876 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.876 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 879 2024-04-19T02:04:04.876 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.877 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:04.877 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.877 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:04.877 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.149201+0000", 2024-04-19T02:04:04.877 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.877 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.877 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.877 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:04.877 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.149201+0000", 2024-04-19T02:04:04.877 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.877 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.877 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.877 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:04.878 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.149202+0000", 2024-04-19T02:04:04.878 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.2399999999999998e-07 2024-04-19T02:04:04.878 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.878 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.878 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:04.878 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.149206+0000", 2024-04-19T02:04:04.878 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.1749999999999998e-06 2024-04-19T02:04:04.878 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.878 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.878 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:04.878 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.149216+0000", 2024-04-19T02:04:04.879 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7170000000000003e-06 2024-04-19T02:04:04.879 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.879 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.879 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.879 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.149223+0000", 2024-04-19T02:04:04.879 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.199e-06 2024-04-19T02:04:04.879 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.879 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.879 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:04.879 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.150569+0000", 2024-04-19T02:04:04.879 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0013462649999999999 2024-04-19T02:04:04.879 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.879 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.880 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:04:04.880 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.150674+0000", 2024-04-19T02:04:04.880 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00010500700000000001 2024-04-19T02:04:04.880 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.880 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.880 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:04.880 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.177475+0000", 2024-04-19T02:04:04.880 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.026801207000000001 2024-04-19T02:04:04.880 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.880 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:04.880 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.880 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.880 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.881 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "osd_op(client.4509.0:880 3.9s0 3:956ba687:::smithi05930959-13:head [read 0~3342336,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-04-19T02:04:04.881 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:04:00.149233+0000", 2024-04-19T02:04:04.881 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 4.6360263149999996, 2024-04-19T02:04:04.881 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.031214571999999999, 2024-04-19T02:04:04.881 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:04.881 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:04.881 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "started", 2024-04-19T02:04:04.881 INFO:teuthology.orchestra.run.smithi081.stdout: "client_info": { 2024-04-19T02:04:04.881 INFO:teuthology.orchestra.run.smithi081.stdout: "client": "client.4509", 2024-04-19T02:04:04.882 INFO:teuthology.orchestra.run.smithi081.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:04.882 INFO:teuthology.orchestra.run.smithi081.stdout: "tid": 880 2024-04-19T02:04:04.882 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.882 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:04.882 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.882 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:04.882 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.149233+0000", 2024-04-19T02:04:04.882 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.882 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.882 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.882 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:04.883 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.149233+0000", 2024-04-19T02:04:04.883 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.883 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.883 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.883 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:04.883 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.149233+0000", 2024-04-19T02:04:04.883 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2e-07 2024-04-19T02:04:04.883 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.883 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.884 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:04.884 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.149237+0000", 2024-04-19T02:04:04.884 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.5619999999999999e-06 2024-04-19T02:04:04.884 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.884 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.884 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:04.884 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.149246+0000", 2024-04-19T02:04:04.884 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7159999999999992e-06 2024-04-19T02:04:04.884 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.884 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.884 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.884 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.149251+0000", 2024-04-19T02:04:04.884 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3589999999999999e-06 2024-04-19T02:04:04.885 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.885 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.885 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:04.885 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.152301+0000", 2024-04-19T02:04:04.885 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.003049838 2024-04-19T02:04:04.885 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.885 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.885 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "started", 2024-04-19T02:04:04.885 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.152426+0000", 2024-04-19T02:04:04.885 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012512499999999999 2024-04-19T02:04:04.885 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.885 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.885 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:04.886 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.180447+0000", 2024-04-19T02:04:04.886 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.028021451999999999 2024-04-19T02:04:04.886 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.886 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:04.886 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.886 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.886 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.886 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.2s3 80/26 ECSubWrite(tid=352, reqid=osd.3.0:352, at_version=80'186, trim_to=0'0, roll_forward_to=79'182))", 2024-04-19T02:04:04.886 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:04:04.162861+0000", 2024-04-19T02:04:04.886 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.62239814900000001, 2024-04-19T02:04:04.886 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.022894962000000001, 2024-04-19T02:04:04.886 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:04.887 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:04.887 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:04.887 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:04.887 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.887 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:04.887 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:04.162861+0000", 2024-04-19T02:04:04.887 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.887 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.887 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.887 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:04.887 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:04.162861+0000", 2024-04-19T02:04:04.887 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.887 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.888 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.888 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:04.888 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:04.162862+0000", 2024-04-19T02:04:04.888 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.4e-07 2024-04-19T02:04:04.888 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.888 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.888 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:04.888 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:04.162904+0000", 2024-04-19T02:04:04.888 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2302999999999998e-05 2024-04-19T02:04:04.888 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.888 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.888 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:04.889 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:04.162926+0000", 2024-04-19T02:04:04.889 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1478999999999999e-05 2024-04-19T02:04:04.889 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.889 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.889 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.889 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:04.162942+0000", 2024-04-19T02:04:04.889 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.6200000000000001e-05 2024-04-19T02:04:04.889 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.889 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.889 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:04.889 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:04.163076+0000", 2024-04-19T02:04:04.889 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00013360299999999999 2024-04-19T02:04:04.889 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.890 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.890 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:04.890 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:04.163098+0000", 2024-04-19T02:04:04.890 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2396e-05 2024-04-19T02:04:04.890 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.890 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.890 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:04.890 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:04.185668+0000", 2024-04-19T02:04:04.890 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.022570296 2024-04-19T02:04:04.890 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.890 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.890 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:04.891 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:04.185756+0000", 2024-04-19T02:04:04.891 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.7744999999999994e-05 2024-04-19T02:04:04.891 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.891 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:04.891 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.891 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.891 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.891 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.bs4 80/26 ECSubWrite(tid=353, reqid=osd.3.0:353, at_version=80'79, trim_to=0'0, roll_forward_to=79'75))", 2024-04-19T02:04:04.891 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:04:04.163606+0000", 2024-04-19T02:04:04.891 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 0.62165329800000002, 2024-04-19T02:04:04.891 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.022264241000000001, 2024-04-19T02:04:04.891 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:04.891 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:04.891 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:04.892 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:04.892 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.892 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:04.892 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:04.163606+0000", 2024-04-19T02:04:04.892 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.892 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.892 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.892 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:04.892 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:04.163606+0000", 2024-04-19T02:04:04.892 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:04.892 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.892 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.892 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:04.893 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:04.163607+0000", 2024-04-19T02:04:04.893 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1999999999999999e-06 2024-04-19T02:04:04.893 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.893 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.893 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:04.893 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:04.163654+0000", 2024-04-19T02:04:04.893 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.6585999999999997e-05 2024-04-19T02:04:04.893 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.893 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.893 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:04.893 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:04.163682+0000", 2024-04-19T02:04:04.893 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8552e-05 2024-04-19T02:04:04.894 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.894 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.894 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:04.894 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:04.163702+0000", 2024-04-19T02:04:04.894 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.965e-05 2024-04-19T02:04:04.894 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.894 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.894 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:04.894 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:04.169062+0000", 2024-04-19T02:04:04.894 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0053605559999999998 2024-04-19T02:04:04.894 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.894 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.895 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:04.895 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:04.169087+0000", 2024-04-19T02:04:04.895 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4358999999999998e-05 2024-04-19T02:04:04.895 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.895 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.895 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:04.895 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:04.185794+0000", 2024-04-19T02:04:04.895 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.016707553 2024-04-19T02:04:04.895 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:04.895 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:04.895 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:04.895 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:04.185870+0000", 2024-04-19T02:04:04.895 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.5784999999999994e-05 2024-04-19T02:04:04.895 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.896 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:04.896 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.896 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:04.896 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:04.896 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:04:04.898 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-19T02:04:04.898 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:04:04.897+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:04:04.998 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:04:04.997+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:04:05.025 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:04:05.025 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:04:05.025 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 0 2024-04-19T02:04:05.025 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:04:05.038 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-19T02:04:05.127 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:04:05.129+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:04:05.157 INFO:tasks.rados.rados.0.smithi059.stdout:361: done (2 left) 2024-04-19T02:04:05.157 INFO:tasks.rados.rados.0.smithi059.stdout:362: done (1 left) 2024-04-19T02:04:05.157 INFO:tasks.rados.rados.0.smithi059.stdout:364: done (0 left) 2024-04-19T02:04:05.158 INFO:tasks.rados.rados.0.smithi059.stdout:365: rollback oid 2 current snap is 30 2024-04-19T02:04:05.158 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 2 to 25 2024-04-19T02:04:05.158 INFO:tasks.rados.rados.0.smithi059.stdout:366: snap_remove snap 27 2024-04-19T02:04:05.191 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-19T02:04:05.200 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:04:05.201+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:04:05.210 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:04:05.210 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:04:05.211 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-04-19T02:04:05.211 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-04-19T02:04:05.211 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:04:05.225 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-19T02:04:05.301 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:04:05.301+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:04:05.396 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:04:05.396 INFO:teuthology.orchestra.run.smithi081.stdout: "size": 20, 2024-04-19T02:04:05.396 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 600, 2024-04-19T02:04:05.396 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [ 2024-04-19T02:04:05.396 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.396 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.4s1 42/26 ECSubWrite(tid=44, reqid=osd.1.0:44, at_version=42'32, trim_to=0'0, roll_forward_to=40'28))", 2024-04-19T02:04:05.396 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.738509+0000", 2024-04-19T02:04:05.396 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 38.657318687999997, 2024-04-19T02:04:05.397 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.01226059, 2024-04-19T02:04:05.397 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:05.397 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:05.397 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:05.397 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:05.397 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.397 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:05.397 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.738509+0000", 2024-04-19T02:04:05.397 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.397 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.397 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.397 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:05.397 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.738509+0000", 2024-04-19T02:04:05.398 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.398 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.398 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.398 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:05.398 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.738511+0000", 2024-04-19T02:04:05.398 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.84e-06 2024-04-19T02:04:05.398 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.398 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.398 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:05.398 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.738565+0000", 2024-04-19T02:04:05.398 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3643000000000001e-05 2024-04-19T02:04:05.398 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.398 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.398 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:05.399 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.738593+0000", 2024-04-19T02:04:05.399 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8810999999999999e-05 2024-04-19T02:04:05.399 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.399 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.399 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:05.399 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.738615+0000", 2024-04-19T02:04:05.399 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1421999999999999e-05 2024-04-19T02:04:05.399 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.399 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.399 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:05.399 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.739435+0000", 2024-04-19T02:04:05.399 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00081972100000000003 2024-04-19T02:04:05.399 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.399 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.400 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:05.400 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.739460+0000", 2024-04-19T02:04:05.400 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5646000000000001e-05 2024-04-19T02:04:05.400 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.400 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.400 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:05.400 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.750672+0000", 2024-04-19T02:04:05.400 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011211563000000001 2024-04-19T02:04:05.400 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.400 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.400 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:05.400 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.750770+0000", 2024-04-19T02:04:05.400 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7943999999999996e-05 2024-04-19T02:04:05.400 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.401 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:05.401 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.401 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.401 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.401 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.ds4 42/26 ECSubWrite(tid=163, reqid=osd.8.0:163, at_version=42'14, trim_to=0'0, roll_forward_to=39'12))", 2024-04-19T02:04:05.401 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.742684+0000", 2024-04-19T02:04:05.401 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 38.653143741000001, 2024-04-19T02:04:05.401 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012960826, 2024-04-19T02:04:05.401 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:05.401 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:05.401 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:05.401 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:05.401 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.401 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:05.401 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.742684+0000", 2024-04-19T02:04:05.402 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.402 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.402 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.402 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:05.402 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.742684+0000", 2024-04-19T02:04:05.402 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.402 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.402 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.402 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:05.402 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.742685+0000", 2024-04-19T02:04:05.402 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.0999999999999998e-07 2024-04-19T02:04:05.402 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.402 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.402 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:05.403 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.742739+0000", 2024-04-19T02:04:05.403 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.4089999999999999e-05 2024-04-19T02:04:05.403 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.403 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.403 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:05.403 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.742770+0000", 2024-04-19T02:04:05.403 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0580999999999997e-05 2024-04-19T02:04:05.403 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.403 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.403 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:05.403 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.742792+0000", 2024-04-19T02:04:05.403 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2985000000000001e-05 2024-04-19T02:04:05.403 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.404 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.404 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:05.404 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.743203+0000", 2024-04-19T02:04:05.404 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000410379 2024-04-19T02:04:05.404 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.404 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.404 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:05.404 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.743230+0000", 2024-04-19T02:04:05.404 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6848000000000001e-05 2024-04-19T02:04:05.404 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.404 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.404 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:05.404 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755545+0000", 2024-04-19T02:04:05.404 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012315609 2024-04-19T02:04:05.405 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.405 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.405 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:05.405 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.755645+0000", 2024-04-19T02:04:05.405 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9623999999999993e-05 2024-04-19T02:04:05.405 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.405 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:05.405 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.405 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.405 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.405 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.2s1 42/26 ECSubWrite(tid=132, reqid=osd.3.0:132, at_version=42'63, trim_to=0'0, roll_forward_to=40'61))", 2024-04-19T02:04:05.405 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.746263+0000", 2024-04-19T02:04:05.405 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 38.649564877000003, 2024-04-19T02:04:05.405 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013143470000000001, 2024-04-19T02:04:05.406 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:05.406 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:05.406 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:05.406 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:05.406 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.406 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:05.406 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746263+0000", 2024-04-19T02:04:05.406 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.406 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.406 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.406 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:05.406 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746263+0000", 2024-04-19T02:04:05.406 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.407 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.407 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.407 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:05.407 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746264+0000", 2024-04-19T02:04:05.407 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9399999999999993e-07 2024-04-19T02:04:05.407 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.407 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.407 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:05.407 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746323+0000", 2024-04-19T02:04:05.407 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8686e-05 2024-04-19T02:04:05.407 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.407 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.407 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:05.407 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746353+0000", 2024-04-19T02:04:05.407 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0394000000000001e-05 2024-04-19T02:04:05.407 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.407 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.408 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:05.408 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746375+0000", 2024-04-19T02:04:05.408 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1934999999999999e-05 2024-04-19T02:04:05.408 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.408 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.408 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:05.408 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746786+0000", 2024-04-19T02:04:05.408 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00041093199999999997 2024-04-19T02:04:05.408 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.408 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.410 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:05.410 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746812+0000", 2024-04-19T02:04:05.410 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6392000000000001e-05 2024-04-19T02:04:05.410 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.410 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.410 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:05.411 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.759279+0000", 2024-04-19T02:04:05.411 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012466589 2024-04-19T02:04:05.411 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.411 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.411 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:05.411 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.759406+0000", 2024-04-19T02:04:05.411 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00012754800000000001 2024-04-19T02:04:05.411 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.411 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:05.411 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.411 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.411 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.411 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.8s4 42/26 ECSubWrite(tid=71, reqid=osd.4.0:71, at_version=42'33, trim_to=0'0, roll_forward_to=37'31))", 2024-04-19T02:04:05.411 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.746410+0000", 2024-04-19T02:04:05.411 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 38.649417872999997, 2024-04-19T02:04:05.411 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012849090000000001, 2024-04-19T02:04:05.412 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:05.412 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:05.412 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:05.412 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:05.412 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.412 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:05.412 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746410+0000", 2024-04-19T02:04:05.413 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.413 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.413 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.413 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:05.413 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746410+0000", 2024-04-19T02:04:05.413 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.413 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.413 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.413 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:05.413 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746411+0000", 2024-04-19T02:04:05.413 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9000000000000005e-07 2024-04-19T02:04:05.413 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.413 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.413 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:05.414 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746467+0000", 2024-04-19T02:04:05.414 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6149000000000001e-05 2024-04-19T02:04:05.414 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.414 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.414 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:05.414 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746511+0000", 2024-04-19T02:04:05.414 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4317000000000003e-05 2024-04-19T02:04:05.414 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.414 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.414 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:05.414 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.746535+0000", 2024-04-19T02:04:05.415 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3655e-05 2024-04-19T02:04:05.415 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.415 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.415 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:05.415 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.747876+0000", 2024-04-19T02:04:05.415 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0013412490000000001 2024-04-19T02:04:05.415 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.415 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.415 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:05.415 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.747901+0000", 2024-04-19T02:04:05.415 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4572000000000001e-05 2024-04-19T02:04:05.415 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.415 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.416 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:05.416 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.759159+0000", 2024-04-19T02:04:05.416 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011258351 2024-04-19T02:04:05.416 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.416 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.416 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:05.416 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.759259+0000", 2024-04-19T02:04:05.416 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9807000000000002e-05 2024-04-19T02:04:05.416 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.416 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:05.416 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.416 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.416 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.416 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.as4 42/26 ECSubWrite(tid=82, reqid=osd.11.0:82, at_version=42'34, trim_to=0'0, roll_forward_to=39'32))", 2024-04-19T02:04:05.417 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.749301+0000", 2024-04-19T02:04:05.417 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 38.646526666, 2024-04-19T02:04:05.417 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.015619305999999999, 2024-04-19T02:04:05.417 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:05.417 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:05.417 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:05.417 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:05.417 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.417 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:05.417 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749301+0000", 2024-04-19T02:04:05.417 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.417 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.417 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.417 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:05.418 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749301+0000", 2024-04-19T02:04:05.418 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.418 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.418 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.418 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:05.418 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749302+0000", 2024-04-19T02:04:05.418 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.5000000000000002e-07 2024-04-19T02:04:05.418 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.418 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.418 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:05.418 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749357+0000", 2024-04-19T02:04:05.418 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.5013000000000003e-05 2024-04-19T02:04:05.418 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.418 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.419 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:05.419 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749391+0000", 2024-04-19T02:04:05.419 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.3821e-05 2024-04-19T02:04:05.419 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.419 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.419 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:05.419 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749414+0000", 2024-04-19T02:04:05.419 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3584999999999999e-05 2024-04-19T02:04:05.419 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.419 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.419 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:05.419 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.750335+0000", 2024-04-19T02:04:05.419 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00092032799999999999 2024-04-19T02:04:05.419 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.419 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.419 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:05.420 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.750362+0000", 2024-04-19T02:04:05.420 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7217999999999999e-05 2024-04-19T02:04:05.420 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.420 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.420 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:05.420 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.764821+0000", 2024-04-19T02:04:05.420 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014459520999999999 2024-04-19T02:04:05.420 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.420 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.420 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:05.420 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.764921+0000", 2024-04-19T02:04:05.420 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.9170000000000001e-05 2024-04-19T02:04:05.420 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.420 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:05.421 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.421 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.421 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.421 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.5s7 42/26 ECSubWrite(tid=51, reqid=osd.10.0:51, at_version=42'27, trim_to=0'0, roll_forward_to=39'25))", 2024-04-19T02:04:05.421 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.749382+0000", 2024-04-19T02:04:05.421 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 38.646445428, 2024-04-19T02:04:05.421 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010066832, 2024-04-19T02:04:05.421 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:05.421 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:05.421 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:05.421 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:05.421 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.421 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:05.421 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749382+0000", 2024-04-19T02:04:05.421 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.421 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.422 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.422 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:05.422 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749382+0000", 2024-04-19T02:04:05.422 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.422 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.422 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.422 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:05.422 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749383+0000", 2024-04-19T02:04:05.422 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7000000000000003e-07 2024-04-19T02:04:05.422 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.422 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.422 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:05.422 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749433+0000", 2024-04-19T02:04:05.423 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.9812999999999999e-05 2024-04-19T02:04:05.423 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.423 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.423 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:05.423 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749459+0000", 2024-04-19T02:04:05.423 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5565e-05 2024-04-19T02:04:05.423 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.423 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.423 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:05.423 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749478+0000", 2024-04-19T02:04:05.423 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9216e-05 2024-04-19T02:04:05.423 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.423 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.423 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:05.424 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749657+0000", 2024-04-19T02:04:05.424 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00017872799999999999 2024-04-19T02:04:05.424 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.424 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.424 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:05.424 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749682+0000", 2024-04-19T02:04:05.424 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5734000000000001e-05 2024-04-19T02:04:05.424 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.424 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.424 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:05.424 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.759360+0000", 2024-04-19T02:04:05.424 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0096778550000000008 2024-04-19T02:04:05.424 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.424 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.424 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:05.425 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.759449+0000", 2024-04-19T02:04:05.425 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.8950999999999996e-05 2024-04-19T02:04:05.425 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.425 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:05.425 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.425 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.425 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.425 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.2s1 42/26 ECSubWrite(tid=134, reqid=osd.3.0:134, at_version=42'65, trim_to=0'0, roll_forward_to=40'61))", 2024-04-19T02:04:05.425 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.749777+0000", 2024-04-19T02:04:05.425 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 38.646050864000003, 2024-04-19T02:04:05.425 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.015233401000000001, 2024-04-19T02:04:05.425 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:05.425 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:05.425 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:05.426 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:05.426 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.426 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:05.426 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749777+0000", 2024-04-19T02:04:05.426 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.426 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.426 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.426 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:05.426 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749777+0000", 2024-04-19T02:04:05.426 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.426 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.426 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.427 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:05.427 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749778+0000", 2024-04-19T02:04:05.427 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.336e-06 2024-04-19T02:04:05.427 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.427 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.427 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:05.427 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749823+0000", 2024-04-19T02:04:05.427 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4277000000000003e-05 2024-04-19T02:04:05.427 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.427 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.427 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:05.427 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749845+0000", 2024-04-19T02:04:05.427 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2677999999999999e-05 2024-04-19T02:04:05.427 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.427 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.428 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:05.428 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.749864+0000", 2024-04-19T02:04:05.428 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8913e-05 2024-04-19T02:04:05.428 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.428 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.428 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:05.428 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.753987+0000", 2024-04-19T02:04:05.428 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0041232070000000003 2024-04-19T02:04:05.428 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.428 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.428 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:05.428 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.754010+0000", 2024-04-19T02:04:05.428 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2792000000000001e-05 2024-04-19T02:04:05.428 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.428 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.429 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:05.429 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.764933+0000", 2024-04-19T02:04:05.429 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.01092277 2024-04-19T02:04:05.429 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.429 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.429 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:05.429 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.765010+0000", 2024-04-19T02:04:05.429 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7428000000000005e-05 2024-04-19T02:04:05.429 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.429 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:05.429 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.429 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.429 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.429 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.es2 42/26 ECSubWrite(tid=32, reqid=osd.7.0:32, at_version=42'25, trim_to=0'0, roll_forward_to=40'23))", 2024-04-19T02:04:05.429 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.752167+0000", 2024-04-19T02:04:05.430 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 38.643660879999999, 2024-04-19T02:04:05.430 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012865329, 2024-04-19T02:04:05.430 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:05.430 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:05.430 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:05.430 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:05.430 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.430 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:05.430 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.752167+0000", 2024-04-19T02:04:05.430 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.430 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.430 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.430 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:05.430 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.752167+0000", 2024-04-19T02:04:05.430 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.431 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.431 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.431 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:05.431 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.752168+0000", 2024-04-19T02:04:05.431 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6700000000000003e-07 2024-04-19T02:04:05.431 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.431 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.431 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:05.431 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.752240+0000", 2024-04-19T02:04:05.431 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1988000000000003e-05 2024-04-19T02:04:05.431 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.431 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.431 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:05.431 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.752265+0000", 2024-04-19T02:04:05.431 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5626000000000002e-05 2024-04-19T02:04:05.432 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.432 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.432 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:05.432 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.752287+0000", 2024-04-19T02:04:05.432 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2045e-05 2024-04-19T02:04:05.432 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.432 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.432 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:05.432 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.752546+0000", 2024-04-19T02:04:05.432 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000258342 2024-04-19T02:04:05.432 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.432 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.432 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:05.432 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.752571+0000", 2024-04-19T02:04:05.433 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4941999999999999e-05 2024-04-19T02:04:05.433 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.433 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.433 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:05.433 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.764965+0000", 2024-04-19T02:04:05.433 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.012394196999999999 2024-04-19T02:04:05.433 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.433 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.433 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:05.433 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.765032+0000", 2024-04-19T02:04:05.433 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7621999999999997e-05 2024-04-19T02:04:05.433 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.433 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:05.433 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.433 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.434 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.434 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.4s1 42/26 ECSubWrite(tid=45, reqid=osd.1.0:44, at_version=0'0, trim_to=0'0, roll_forward_to=42'32))", 2024-04-19T02:04:05.434 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:26.754514+0000", 2024-04-19T02:04:05.434 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 38.641314184999999, 2024-04-19T02:04:05.434 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011039706, 2024-04-19T02:04:05.434 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:05.434 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:05.434 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:05.434 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:05.434 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.434 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:05.434 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.754514+0000", 2024-04-19T02:04:05.434 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.434 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.435 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.435 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:05.435 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.754514+0000", 2024-04-19T02:04:05.435 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.435 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.435 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.435 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:05.435 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.754516+0000", 2024-04-19T02:04:05.435 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8560000000000001e-06 2024-04-19T02:04:05.435 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.435 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.435 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:05.435 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.754544+0000", 2024-04-19T02:04:05.435 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.8248000000000001e-05 2024-04-19T02:04:05.435 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.436 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.436 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:05.436 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.754574+0000", 2024-04-19T02:04:05.436 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9935000000000001e-05 2024-04-19T02:04:05.436 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.436 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.436 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:05.436 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.754595+0000", 2024-04-19T02:04:05.436 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.1596000000000001e-05 2024-04-19T02:04:05.436 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.436 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.436 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:05.436 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.754758+0000", 2024-04-19T02:04:05.436 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016278500000000001 2024-04-19T02:04:05.436 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.437 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.437 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:05.437 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.754784+0000", 2024-04-19T02:04:05.437 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5698e-05 2024-04-19T02:04:05.437 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.437 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.437 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:05.437 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.765464+0000", 2024-04-19T02:04:05.437 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010680044 2024-04-19T02:04:05.437 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.437 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.437 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:05.437 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:26.765553+0000", 2024-04-19T02:04:05.437 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.9543999999999995e-05 2024-04-19T02:04:05.437 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.437 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:05.438 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.438 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.438 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.438 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.as4 46/26 ECSubWrite(tid=103, reqid=osd.11.0:103, at_version=46'49, trim_to=0'0, roll_forward_to=45'45))", 2024-04-19T02:04:05.438 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.067936+0000", 2024-04-19T02:04:05.438 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 35.327891477999998, 2024-04-19T02:04:05.438 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.017806114000000001, 2024-04-19T02:04:05.438 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:05.438 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:05.438 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:05.438 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:05.438 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.438 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:05.438 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.067936+0000", 2024-04-19T02:04:05.439 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.439 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.439 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.439 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:05.439 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.067936+0000", 2024-04-19T02:04:05.439 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.439 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.439 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.439 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:05.439 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.067938+0000", 2024-04-19T02:04:05.439 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.313e-06 2024-04-19T02:04:05.439 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.439 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.439 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:05.439 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.067990+0000", 2024-04-19T02:04:05.440 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.2490000000000001e-05 2024-04-19T02:04:05.440 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.440 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.440 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:05.440 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.068014+0000", 2024-04-19T02:04:05.440 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4172000000000002e-05 2024-04-19T02:04:05.440 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.440 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.440 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:05.440 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.068034+0000", 2024-04-19T02:04:05.440 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9471999999999999e-05 2024-04-19T02:04:05.440 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.440 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.440 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:05.440 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.069268+0000", 2024-04-19T02:04:05.441 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001234165 2024-04-19T02:04:05.441 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.441 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.441 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:05.441 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.069293+0000", 2024-04-19T02:04:05.441 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4702000000000002e-05 2024-04-19T02:04:05.441 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.441 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.441 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:05.441 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.085653+0000", 2024-04-19T02:04:05.441 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.016360535999999998 2024-04-19T02:04:05.441 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.441 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.441 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:05.441 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.085743+0000", 2024-04-19T02:04:05.442 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.9264000000000005e-05 2024-04-19T02:04:05.442 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.442 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:05.442 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.442 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.442 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.442 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.1s3 46/26 ECSubWrite(tid=20, reqid=client.4509.0:415, at_version=46'8, trim_to=0'0, roll_forward_to=29'6))", 2024-04-19T02:04:05.442 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.077639+0000", 2024-04-19T02:04:05.442 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 35.318188808000002, 2024-04-19T02:04:05.442 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010858876999999999, 2024-04-19T02:04:05.442 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:05.442 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:05.442 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:05.442 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:05.443 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.443 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:05.443 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.077639+0000", 2024-04-19T02:04:05.443 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.443 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.443 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.443 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:05.443 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.077639+0000", 2024-04-19T02:04:05.443 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.443 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.443 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.443 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:05.443 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.077640+0000", 2024-04-19T02:04:05.443 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.1999999999999999e-07 2024-04-19T02:04:05.443 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.444 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.444 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:05.444 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078039+0000", 2024-04-19T02:04:05.444 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00039909899999999998 2024-04-19T02:04:05.444 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.444 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.444 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:05.444 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078071+0000", 2024-04-19T02:04:05.444 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.1987999999999999e-05 2024-04-19T02:04:05.444 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.444 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.444 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:05.444 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078093+0000", 2024-04-19T02:04:05.444 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2086000000000002e-05 2024-04-19T02:04:05.445 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.445 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.445 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:05.445 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078730+0000", 2024-04-19T02:04:05.445 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00063664699999999995 2024-04-19T02:04:05.445 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.445 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.445 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:05.445 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.078755+0000", 2024-04-19T02:04:05.445 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5029e-05 2024-04-19T02:04:05.445 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.445 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.445 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:05.445 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.088431+0000", 2024-04-19T02:04:05.445 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0096761820000000002 2024-04-19T02:04:05.446 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.446 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.446 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:05.446 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.088498+0000", 2024-04-19T02:04:05.446 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.7125999999999994e-05 2024-04-19T02:04:05.446 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.446 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:05.446 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.446 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.446 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.446 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.2s1 46/26 ECSubWrite(tid=163, reqid=client.4509.0:425, at_version=46'78, trim_to=0'0, roll_forward_to=46'75))", 2024-04-19T02:04:05.446 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.196489+0000", 2024-04-19T02:04:05.446 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 35.199338859000001, 2024-04-19T02:04:05.446 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013644882000000001, 2024-04-19T02:04:05.446 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:05.447 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:05.447 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:05.447 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:05.447 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.447 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:05.447 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196489+0000", 2024-04-19T02:04:05.447 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.447 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.447 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.447 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:05.447 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196489+0000", 2024-04-19T02:04:05.447 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.447 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.447 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.447 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:05.447 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196491+0000", 2024-04-19T02:04:05.448 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.4269999999999998e-06 2024-04-19T02:04:05.448 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.448 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.448 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:05.448 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196543+0000", 2024-04-19T02:04:05.448 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.1112999999999997e-05 2024-04-19T02:04:05.448 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.448 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.448 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:05.448 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196607+0000", 2024-04-19T02:04:05.448 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.4183000000000005e-05 2024-04-19T02:04:05.448 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.448 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.448 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:05.448 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196634+0000", 2024-04-19T02:04:05.448 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6968e-05 2024-04-19T02:04:05.449 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.449 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.449 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:05.449 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.198568+0000", 2024-04-19T02:04:05.449 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0019338129999999999 2024-04-19T02:04:05.449 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.449 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.449 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:05.449 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.198591+0000", 2024-04-19T02:04:05.449 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.3898000000000001e-05 2024-04-19T02:04:05.449 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.449 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.449 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:05.449 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.210049+0000", 2024-04-19T02:04:05.450 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011457355000000001 2024-04-19T02:04:05.450 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.450 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.450 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:05.450 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.210134+0000", 2024-04-19T02:04:05.450 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.5124999999999996e-05 2024-04-19T02:04:05.450 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.450 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:05.450 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.450 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.450 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.450 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.ds4 46/26 ECSubWrite(tid=190, reqid=client.4509.0:428, at_version=46'37, trim_to=0'0, roll_forward_to=46'35))", 2024-04-19T02:04:05.450 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.196834+0000", 2024-04-19T02:04:05.451 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 35.198993575999999, 2024-04-19T02:04:05.451 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011503825000000001, 2024-04-19T02:04:05.451 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:05.451 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:05.451 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:05.451 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:05.451 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.451 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:05.451 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196834+0000", 2024-04-19T02:04:05.451 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.451 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.451 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.451 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:05.451 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196834+0000", 2024-04-19T02:04:05.451 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.452 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.452 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.452 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:05.452 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196835+0000", 2024-04-19T02:04:05.452 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.13e-06 2024-04-19T02:04:05.452 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.452 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.452 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:05.452 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196895+0000", 2024-04-19T02:04:05.452 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.9481999999999997e-05 2024-04-19T02:04:05.452 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.452 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.452 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:05.452 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196922+0000", 2024-04-19T02:04:05.453 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.6639e-05 2024-04-19T02:04:05.453 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.453 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.453 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:05.453 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.196942+0000", 2024-04-19T02:04:05.453 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0931999999999999e-05 2024-04-19T02:04:05.453 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.453 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.453 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:05.453 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.197738+0000", 2024-04-19T02:04:05.453 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00079601499999999996 2024-04-19T02:04:05.453 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.453 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.453 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:05.453 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.197811+0000", 2024-04-19T02:04:05.454 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.2916000000000005e-05 2024-04-19T02:04:05.454 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.454 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.454 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:05.454 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.208200+0000", 2024-04-19T02:04:05.454 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010388246 2024-04-19T02:04:05.454 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.454 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.454 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:05.454 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.208338+0000", 2024-04-19T02:04:05.454 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.000138465 2024-04-19T02:04:05.454 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.454 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:05.454 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.455 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.455 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.455 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.2s1 46/26 ECSubWrite(tid=164, reqid=client.4509.0:426, at_version=46'79, trim_to=0'0, roll_forward_to=46'75))", 2024-04-19T02:04:05.455 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:30.199712+0000", 2024-04-19T02:04:05.455 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 35.196116171, 2024-04-19T02:04:05.455 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010489036, 2024-04-19T02:04:05.455 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:05.455 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:05.455 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:05.455 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:05.455 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.457 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:05.457 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.199712+0000", 2024-04-19T02:04:05.457 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.457 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.457 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.458 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:05.458 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.199712+0000", 2024-04-19T02:04:05.458 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.458 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.458 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.458 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:05.458 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.199713+0000", 2024-04-19T02:04:05.458 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7630000000000001e-06 2024-04-19T02:04:05.458 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.458 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.458 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:05.458 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.199758+0000", 2024-04-19T02:04:05.458 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4969999999999998e-05 2024-04-19T02:04:05.458 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.459 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.459 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:05.459 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.199784+0000", 2024-04-19T02:04:05.459 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5846e-05 2024-04-19T02:04:05.459 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.459 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.459 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:05.459 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.199805+0000", 2024-04-19T02:04:05.459 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.0271999999999998e-05 2024-04-19T02:04:05.459 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.459 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.459 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:05.459 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.199970+0000", 2024-04-19T02:04:05.459 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00016560899999999999 2024-04-19T02:04:05.459 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.459 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.460 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:05.460 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.199993+0000", 2024-04-19T02:04:05.460 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2549e-05 2024-04-19T02:04:05.460 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.460 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.460 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:05.460 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.210142+0000", 2024-04-19T02:04:05.460 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010149768 2024-04-19T02:04:05.460 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.460 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.460 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:05.460 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:30.210201+0000", 2024-04-19T02:04:05.460 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.8258999999999999e-05 2024-04-19T02:04:05.460 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.460 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:05.461 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.461 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.461 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.461 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.1s3 47/26 ECSubWrite(tid=30, reqid=osd.0.0:30, at_version=47'17, trim_to=0'0, roll_forward_to=46'13))", 2024-04-19T02:04:05.461 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:31.085565+0000", 2024-04-19T02:04:05.461 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 34.310262756999997, 2024-04-19T02:04:05.461 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010262217000000001, 2024-04-19T02:04:05.461 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:05.461 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:05.461 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:05.461 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:05.461 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.461 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:05.461 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085565+0000", 2024-04-19T02:04:05.461 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.462 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.462 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.462 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:05.462 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085565+0000", 2024-04-19T02:04:05.462 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.462 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.462 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.462 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:05.462 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085566+0000", 2024-04-19T02:04:05.462 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.33e-07 2024-04-19T02:04:05.462 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.462 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.462 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:05.462 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085617+0000", 2024-04-19T02:04:05.463 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.0487e-05 2024-04-19T02:04:05.463 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.463 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.463 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:05.463 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085639+0000", 2024-04-19T02:04:05.463 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2866000000000001e-05 2024-04-19T02:04:05.463 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.463 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.463 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:05.463 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.085658+0000", 2024-04-19T02:04:05.463 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8575000000000001e-05 2024-04-19T02:04:05.463 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.463 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.463 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:05.463 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.090331+0000", 2024-04-19T02:04:05.464 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0046729650000000003 2024-04-19T02:04:05.464 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.464 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.464 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:05.464 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.090358+0000", 2024-04-19T02:04:05.464 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7424999999999999e-05 2024-04-19T02:04:05.464 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.464 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.464 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:05.464 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.095729+0000", 2024-04-19T02:04:05.464 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0053710989999999998 2024-04-19T02:04:05.464 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.464 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.464 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:05.464 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:31.095827+0000", 2024-04-19T02:04:05.465 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.7867000000000003e-05 2024-04-19T02:04:05.465 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.465 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:05.465 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.465 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.465 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.465 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.8s4 66/26 ECSubWrite(tid=150, reqid=client.4509.0:722, at_version=66'65, trim_to=0'0, roll_forward_to=57'63))", 2024-04-19T02:04:05.465 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:03:50.060860+0000", 2024-04-19T02:04:05.465 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 15.334968108, 2024-04-19T02:04:05.465 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014413267, 2024-04-19T02:04:05.465 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:05.465 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:05.465 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:05.465 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:05.466 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.466 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:05.466 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.060860+0000", 2024-04-19T02:04:05.466 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.466 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.466 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.466 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:05.466 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.060860+0000", 2024-04-19T02:04:05.466 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.466 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.466 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.466 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:05.466 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.060860+0000", 2024-04-19T02:04:05.466 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0299999999999999e-07 2024-04-19T02:04:05.467 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.467 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.467 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:05.467 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.060914+0000", 2024-04-19T02:04:05.467 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.3627e-05 2024-04-19T02:04:05.467 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.467 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.467 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:05.467 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.060943+0000", 2024-04-19T02:04:05.467 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.9461000000000001e-05 2024-04-19T02:04:05.467 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.467 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.467 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:05.467 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.060963+0000", 2024-04-19T02:04:05.467 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.9168e-05 2024-04-19T02:04:05.468 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.468 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.468 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:05.468 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.061169+0000", 2024-04-19T02:04:05.468 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00020598 2024-04-19T02:04:05.468 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.468 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.468 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:05.468 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.061196+0000", 2024-04-19T02:04:05.468 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7631999999999999e-05 2024-04-19T02:04:05.468 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.468 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.468 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:05.468 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.075230+0000", 2024-04-19T02:04:05.468 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.014033990999999999 2024-04-19T02:04:05.469 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.469 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.469 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:05.469 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:03:50.075273+0000", 2024-04-19T02:04:05.469 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.2805000000000001e-05 2024-04-19T02:04:05.469 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.469 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:05.469 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.469 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.469 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.469 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.4s1 76/26 ECSubWrite(tid=121, reqid=client.4509.0:877, at_version=76'101, trim_to=0'0, roll_forward_to=76'97))", 2024-04-19T02:04:05.469 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:04:00.153128+0000", 2024-04-19T02:04:05.469 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 5.2426998769999997, 2024-04-19T02:04:05.469 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011393932000000001, 2024-04-19T02:04:05.469 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:05.470 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:05.470 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:05.470 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:05.470 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.470 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:05.470 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.153128+0000", 2024-04-19T02:04:05.470 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.470 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.470 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.470 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:05.470 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.153128+0000", 2024-04-19T02:04:05.470 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.470 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.470 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.470 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:05.471 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.153129+0000", 2024-04-19T02:04:05.471 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 5.6400000000000002e-07 2024-04-19T02:04:05.471 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.471 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.471 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:05.471 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.153147+0000", 2024-04-19T02:04:05.471 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.8751000000000001e-05 2024-04-19T02:04:05.471 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.471 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.471 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:05.471 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.153157+0000", 2024-04-19T02:04:05.471 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 9.577e-06 2024-04-19T02:04:05.471 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.471 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.472 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:05.472 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.153165+0000", 2024-04-19T02:04:05.472 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 7.7079999999999997e-06 2024-04-19T02:04:05.472 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.472 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.472 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:05.472 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.156151+0000", 2024-04-19T02:04:05.472 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0029867489999999999 2024-04-19T02:04:05.472 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.472 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.472 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:05.472 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.156164+0000", 2024-04-19T02:04:05.472 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.3002e-05 2024-04-19T02:04:05.472 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.473 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.473 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:05.473 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.164461+0000", 2024-04-19T02:04:05.473 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0082967160000000009 2024-04-19T02:04:05.473 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.473 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.473 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:05.473 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.164522+0000", 2024-04-19T02:04:05.473 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.0865000000000001e-05 2024-04-19T02:04:05.473 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.473 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:05.473 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.473 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.473 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.473 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.fs4 76/26 ECSubWrite(tid=184, reqid=client.4509.0:872, at_version=76'45, trim_to=0'0, roll_forward_to=60'43))", 2024-04-19T02:04:05.474 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:04:00.153179+0000", 2024-04-19T02:04:05.474 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 5.2426488139999998, 2024-04-19T02:04:05.474 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.013406881000000001, 2024-04-19T02:04:05.474 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:05.474 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:05.474 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:05.474 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:05.474 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.474 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:05.474 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.153179+0000", 2024-04-19T02:04:05.474 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.474 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.474 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.474 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:05.475 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.153179+0000", 2024-04-19T02:04:05.475 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.475 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.475 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.475 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:05.475 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.153180+0000", 2024-04-19T02:04:05.475 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 8.1200000000000002e-07 2024-04-19T02:04:05.475 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.475 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.475 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:05.475 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.154396+0000", 2024-04-19T02:04:05.475 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001215825 2024-04-19T02:04:05.475 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.475 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.475 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:05.476 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.154426+0000", 2024-04-19T02:04:05.476 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.0029e-05 2024-04-19T02:04:05.476 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.476 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.476 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:05.476 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.154448+0000", 2024-04-19T02:04:05.476 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2163000000000001e-05 2024-04-19T02:04:05.476 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.476 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.476 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:05.476 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.154806+0000", 2024-04-19T02:04:05.476 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00035782000000000001 2024-04-19T02:04:05.476 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.476 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.476 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:05.477 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.154828+0000", 2024-04-19T02:04:05.477 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.2589e-05 2024-04-19T02:04:05.477 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.477 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.477 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:05.477 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.166519+0000", 2024-04-19T02:04:05.477 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.011691012000000001 2024-04-19T02:04:05.477 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.477 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.477 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:05.477 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.166586+0000", 2024-04-19T02:04:05.477 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.6631000000000006e-05 2024-04-19T02:04:05.477 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.477 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:05.478 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.478 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.478 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.478 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.2s1 76/26 ECSubWrite(tid=309, reqid=client.4509.0:887, at_version=76'153, trim_to=0'0, roll_forward_to=76'151))", 2024-04-19T02:04:05.478 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:04:00.156300+0000", 2024-04-19T02:04:05.478 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 5.2395275379999999, 2024-04-19T02:04:05.478 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.010294920000000001, 2024-04-19T02:04:05.478 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:05.478 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:05.478 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:05.478 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:05.478 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.478 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:05.478 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.156300+0000", 2024-04-19T02:04:05.479 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.479 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.479 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.479 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:05.479 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.156300+0000", 2024-04-19T02:04:05.479 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.479 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.479 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.479 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:05.479 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.156302+0000", 2024-04-19T02:04:05.479 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.4589999999999999e-06 2024-04-19T02:04:05.479 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.479 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.479 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:05.479 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.156337+0000", 2024-04-19T02:04:05.480 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 3.4910999999999998e-05 2024-04-19T02:04:05.480 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.480 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.480 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:05.480 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.156354+0000", 2024-04-19T02:04:05.480 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7124000000000001e-05 2024-04-19T02:04:05.480 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.480 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.480 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:05.480 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.156366+0000", 2024-04-19T02:04:05.480 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.2026e-05 2024-04-19T02:04:05.480 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.480 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.480 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:05.481 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.159597+0000", 2024-04-19T02:04:05.481 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0032307939999999999 2024-04-19T02:04:05.481 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.481 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.481 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:05.481 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.159624+0000", 2024-04-19T02:04:05.481 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.7212e-05 2024-04-19T02:04:05.481 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.481 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.481 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:05.481 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.166551+0000", 2024-04-19T02:04:05.481 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0069269220000000003 2024-04-19T02:04:05.481 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.481 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.481 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:05.482 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.166595+0000", 2024-04-19T02:04:05.482 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 4.4471999999999997e-05 2024-04-19T02:04:05.482 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.482 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:05.482 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.482 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.482 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.482 INFO:teuthology.orchestra.run.smithi081.stdout: "description": "MOSDECSubOpWrite(3.8s4 76/26 ECSubWrite(tid=173, reqid=client.4509.0:870, at_version=76'79, trim_to=0'0, roll_forward_to=73'77))", 2024-04-19T02:04:05.482 INFO:teuthology.orchestra.run.smithi081.stdout: "initiated_at": "2024-04-19T02:04:00.159231+0000", 2024-04-19T02:04:05.482 INFO:teuthology.orchestra.run.smithi081.stdout: "age": 5.2365966589999999, 2024-04-19T02:04:05.482 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.01194738, 2024-04-19T02:04:05.482 INFO:teuthology.orchestra.run.smithi081.stdout: "continuous": false, 2024-04-19T02:04:05.482 INFO:teuthology.orchestra.run.smithi081.stdout: "type_data": { 2024-04-19T02:04:05.483 INFO:teuthology.orchestra.run.smithi081.stdout: "flag_point": "reached pg", 2024-04-19T02:04:05.483 INFO:teuthology.orchestra.run.smithi081.stdout: "events": [ 2024-04-19T02:04:05.483 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.483 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "initiated", 2024-04-19T02:04:05.483 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.159231+0000", 2024-04-19T02:04:05.483 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.483 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.483 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.483 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "header_read", 2024-04-19T02:04:05.483 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.159231+0000", 2024-04-19T02:04:05.483 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0 2024-04-19T02:04:05.483 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.483 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.483 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "throttled", 2024-04-19T02:04:05.484 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.159232+0000", 2024-04-19T02:04:05.484 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.0809999999999999e-06 2024-04-19T02:04:05.484 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.484 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.484 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "all_read", 2024-04-19T02:04:05.484 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.160155+0000", 2024-04-19T02:04:05.484 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.00092298499999999997 2024-04-19T02:04:05.484 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.484 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.484 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "dispatched", 2024-04-19T02:04:05.484 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.160173+0000", 2024-04-19T02:04:05.484 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.7229000000000001e-05 2024-04-19T02:04:05.484 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.484 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.484 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "queued_for_pg", 2024-04-19T02:04:05.485 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.160184+0000", 2024-04-19T02:04:05.485 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 1.1277999999999999e-05 2024-04-19T02:04:05.485 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.485 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.485 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "reached_pg", 2024-04-19T02:04:05.485 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.161194+0000", 2024-04-19T02:04:05.485 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.001009767 2024-04-19T02:04:05.485 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.485 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.485 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_started", 2024-04-19T02:04:05.486 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.161219+0000", 2024-04-19T02:04:05.486 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 2.5278000000000001e-05 2024-04-19T02:04:05.486 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.486 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.486 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "sub_op_committed", 2024-04-19T02:04:05.486 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.171109+0000", 2024-04-19T02:04:05.486 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 0.0098906159999999996 2024-04-19T02:04:05.486 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-19T02:04:05.486 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-19T02:04:05.486 INFO:teuthology.orchestra.run.smithi081.stdout: "event": "done", 2024-04-19T02:04:05.486 INFO:teuthology.orchestra.run.smithi081.stdout: "time": "2024-04-19T02:04:00.171179+0000", 2024-04-19T02:04:05.486 INFO:teuthology.orchestra.run.smithi081.stdout: "duration": 6.9146000000000005e-05 2024-04-19T02:04:05.486 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.486 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:05.487 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.487 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-19T02:04:05.487 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-19T02:04:05.487 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:04:05.488 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:04:05.489+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:04:05.489 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-19T02:04:05.589 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:04:05.589+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:04:05.616 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:04:05.617 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:04:05.617 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 0 2024-04-19T02:04:05.617 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:04:05.629 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T02:04:05.630 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-19T02:04:05.690 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:04:05.689+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:04:05.783 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-04-19T02:04:05.784 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T02:04:05.791 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:04:05.793+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:04:05.800 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-04-19T02:04:05.800 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T02:04:05.801 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:04:05.801 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:04:05.801 INFO:teuthology.orchestra.run.smithi081.stdout: "complaint_time": 30, 2024-04-19T02:04:05.801 INFO:teuthology.orchestra.run.smithi081.stdout: "num_blocked_ops": 0 2024-04-19T02:04:05.801 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:04:05.815 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-19T02:04:05.816 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-04-19T02:04:05.817 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T02:04:05.834 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-04-19T02:04:05.834 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T02:04:05.852 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-04-19T02:04:05.853 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T02:04:05.870 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-04-19T02:04:05.870 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T02:04:05.889 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-04-19T02:04:05.889 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T02:04:05.891 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:04:05.893+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:04:05.908 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-04-19T02:04:05.908 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T02:04:05.930 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-04-19T02:04:05.930 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T02:04:05.939 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:05.939 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:05.952 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-19T02:04:05.953 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-04-19T02:04:05.953 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T02:04:05.975 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-04-19T02:04:05.975 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T02:04:05.993 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:04:05.993+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:04:05.998 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-04-19T02:04:05.998 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T02:04:06.094 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:04:06.097+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:04:06.120 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:06.120 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:06.135 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-19T02:04:06.156 INFO:teuthology.orchestra.run.smithi059.stderr:nodeep-scrub is unset 2024-04-19T02:04:06.162 INFO:tasks.rados.rados.0.smithi059.stdout:366: done (1 left) 2024-04-19T02:04:06.163 INFO:tasks.rados.rados.0.smithi059.stdout:367: rmattr oid 49 current snap is 30 2024-04-19T02:04:06.163 INFO:tasks.rados.rados.0.smithi059.stdout:367: done (1 left) 2024-04-19T02:04:06.163 INFO:tasks.rados.rados.0.smithi059.stdout:368: snap_create 2024-04-19T02:04:06.163 INFO:tasks.rados.rados.0.smithi059.stdout:365: finishing rollback tid 0 to smithi05930959-2 2024-04-19T02:04:06.163 INFO:tasks.rados.rados.0.smithi059.stdout:365: finishing rollback tid 1 to smithi05930959-2 2024-04-19T02:04:06.163 INFO:tasks.rados.rados.0.smithi059.stdout:365: finishing rollback tid 2 to smithi05930959-2 2024-04-19T02:04:06.163 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 2 v 37 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-19T02:04:06.195 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:04:06.197+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:04:06.270 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:06.270 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:06.284 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-19T02:04:06.327 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:04:06.329+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:04:06.397 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:04:06.397+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:04:06.450 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:06.450 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:06.464 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-19T02:04:06.498 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:04:06.497+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:04:06.595 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:06.595 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:06.598 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:04:06.597+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:04:06.608 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-19T02:04:06.700 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:04:06.701+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:04:06.728 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:06.728 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:06.743 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-19T02:04:06.801 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:04:06.801+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:04:06.867 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:06.867 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:06.881 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-19T02:04:06.930 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:04:06.933+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:04:07.001 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:07.001 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:07.002 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:04:07.001+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:04:07.015 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-19T02:04:07.057 INFO:tasks.rados.rados.0.smithi059.stdout:365: done (1 left) 2024-04-19T02:04:07.057 INFO:tasks.rados.rados.0.smithi059.stdout:368: done (0 left) 2024-04-19T02:04:07.057 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 31 current snap is 31 2024-04-19T02:04:07.057 INFO:tasks.rados.rados.0.smithi059.stdout:369: seq_num 159 ranges {32768=278528} 2024-04-19T02:04:07.062 INFO:tasks.rados.rados.0.smithi059.stdout:369: writing smithi05930959-31 from 32768 to 311296 tid 1 2024-04-19T02:04:07.062 INFO:tasks.rados.rados.0.smithi059.stdout:370: snap_create 2024-04-19T02:04:07.079 INFO:tasks.rados.rados.0.smithi059.stdout:369: finishing write tid 1 to smithi05930959-31 2024-04-19T02:04:07.080 INFO:tasks.rados.rados.0.smithi059.stdout:369: finishing write tid 2 to smithi05930959-31 2024-04-19T02:04:07.085 INFO:tasks.rados.rados.0.smithi059.stdout:369: finishing write tid 3 to smithi05930959-31 2024-04-19T02:04:07.085 INFO:tasks.rados.rados.0.smithi059.stdout:369: oid 31 updating version 0 to 131 2024-04-19T02:04:07.085 INFO:tasks.rados.rados.0.smithi059.stdout:369: oid 31 updating version 131 to 132 2024-04-19T02:04:07.085 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 31 v 132 (ObjNum 159 snap 31 seq_num 159) dirty exists 2024-04-19T02:04:07.085 INFO:tasks.rados.rados.0.smithi059.stdout:369: left oid 31 (ObjNum 159 snap 31 seq_num 159) 2024-04-19T02:04:07.104 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:04:07.105+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:04:07.185 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:07.185 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:07.199 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-19T02:04:07.250 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:04:07.253+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:04:07.306 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:04:07.305+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:04:07.318 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:07.318 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:07.331 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-19T02:04:07.422 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:04:07.425+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:04:07.502 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:07.502 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:07.507 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:04:07.505+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:04:07.516 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-19T02:04:07.608 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:04:07.609+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:04:07.685 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:07.685 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:07.699 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-19T02:04:07.746 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:04:07.749+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:04:07.812 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:04:07.813+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:04:07.823 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:07.824 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:07.839 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-19T02:04:07.913 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:04:07.913+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:04:08.002 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:08.003 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:08.014 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:04:08.017+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:04:08.017 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-19T02:04:08.019 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T02:04:08.055 INFO:tasks.rados.rados.0.smithi059.stdout:369: done (1 left) 2024-04-19T02:04:08.055 INFO:tasks.rados.rados.0.smithi059.stdout:370: done (0 left) 2024-04-19T02:04:08.055 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 14 current snap is 32 2024-04-19T02:04:08.055 INFO:tasks.rados.rados.0.smithi059.stdout:371: seq_num 160 ranges {2457600=294912} 2024-04-19T02:04:08.074 INFO:tasks.rados.rados.0.smithi059.stdout:371: writing smithi05930959-14 from 2457600 to 2752512 tid 1 2024-04-19T02:04:08.075 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 45 current snap is 32 2024-04-19T02:04:08.075 INFO:tasks.rados.rados.0.smithi059.stdout:372: seq_num 161 ranges {0=311296} 2024-04-19T02:04:08.076 INFO:tasks.rados.rados.0.smithi059.stdout:372: writing smithi05930959-45 from 0 to 311296 tid 1 2024-04-19T02:04:08.077 INFO:tasks.rados.rados.0.smithi059.stdout:373: snap_create 2024-04-19T02:04:08.085 INFO:tasks.rados.rados.0.smithi059.stdout:371: finishing write tid 1 to smithi05930959-14 2024-04-19T02:04:08.085 INFO:tasks.rados.rados.0.smithi059.stdout:371: finishing write tid 2 to smithi05930959-14 2024-04-19T02:04:08.090 INFO:tasks.rados.rados.0.smithi059.stdout:371: finishing write tid 3 to smithi05930959-14 2024-04-19T02:04:08.090 INFO:tasks.rados.rados.0.smithi059.stdout:371: oid 14 updating version 0 to 135 2024-04-19T02:04:08.090 INFO:tasks.rados.rados.0.smithi059.stdout:371: oid 14 version 135 is already newer than 134 2024-04-19T02:04:08.090 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 14 v 135 (ObjNum 160 snap 32 seq_num 160) dirty exists 2024-04-19T02:04:08.090 INFO:tasks.rados.rados.0.smithi059.stdout:371: left oid 14 (ObjNum 160 snap 32 seq_num 160) 2024-04-19T02:04:08.091 INFO:tasks.rados.rados.0.smithi059.stdout:372: finishing write tid 1 to smithi05930959-45 2024-04-19T02:04:08.091 INFO:tasks.rados.rados.0.smithi059.stdout:372: finishing write tid 2 to smithi05930959-45 2024-04-19T02:04:08.095 INFO:tasks.rados.rados.0.smithi059.stdout:372: finishing write tid 3 to smithi05930959-45 2024-04-19T02:04:08.095 INFO:tasks.rados.rados.0.smithi059.stdout:372: oid 45 updating version 0 to 84 2024-04-19T02:04:08.095 INFO:tasks.rados.rados.0.smithi059.stdout:372: oid 45 updating version 84 to 85 2024-04-19T02:04:08.095 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 45 v 85 (ObjNum 161 snap 32 seq_num 161) dirty exists 2024-04-19T02:04:08.095 INFO:tasks.rados.rados.0.smithi059.stdout:372: left oid 45 (ObjNum 161 snap 32 seq_num 161) 2024-04-19T02:04:08.115 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:04:08.117+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:04:08.186 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:08.186 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:08.190 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-04-19T02:04:08.191 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T02:04:08.192 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-19T02:04:08.192 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-19T02:04:08.192 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-04-19T02:04:08.193 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2024-04-19T02:04:08.194 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-19T02:04:08.200 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-19T02:04:08.217 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:04:08.217+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:04:08.317 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:04:08.317+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:04:08.324 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:08.324 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:08.339 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-19T02:04:08.361 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2024-04-19T02:04:08.409 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-04-19T02:04:08.409 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T02:04:08.419 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:04:08.421+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:04:08.428 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-04-19T02:04:08.428 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T02:04:08.447 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-04-19T02:04:08.448 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T02:04:08.457 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:04:08.457 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:04:08.457 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-04-19T02:04:08.457 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:04:08.468 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-04-19T02:04:08.469 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T02:04:08.473 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-19T02:04:08.490 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-04-19T02:04:08.491 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T02:04:08.512 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-04-19T02:04:08.513 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T02:04:08.519 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:04:08.521+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:04:08.538 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-04-19T02:04:08.538 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T02:04:08.566 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-04-19T02:04:08.566 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T02:04:08.575 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2024-04-19T02:04:08.587 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-04-19T02:04:08.587 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T02:04:08.608 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-04-19T02:04:08.609 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T02:04:08.609 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:04:08.609 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:04:08.609 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-04-19T02:04:08.609 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-04-19T02:04:08.609 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:04:08.620 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:04:08.621+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:04:08.624 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-19T02:04:08.630 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-04-19T02:04:08.630 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T02:04:08.721 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:04:08.721+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:04:08.747 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2024-04-19T02:04:08.749 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:04:08.749 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-04-19T02:04:08.749 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-04-19T02:04:08.749 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-04-19T02:04:08.749 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.750 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(osd.9.18:2 3.1s0 3:874b3480:::smithi05930959-36:head [assert-version v6,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:04:08.750 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.775686+0000", 2024-04-19T02:04:08.750 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 43.974767612999997, 2024-04-19T02:04:08.750 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010480552000000001, 2024-04-19T02:04:08.750 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:08.750 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:08.750 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:04:08.751 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:08.751 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.751 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:08.751 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.775686+0000", 2024-04-19T02:04:08.751 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.751 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.751 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.751 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:08.751 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.775686+0000", 2024-04-19T02:04:08.751 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.751 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.751 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.751 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:08.751 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.775687+0000", 2024-04-19T02:04:08.752 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3659999999999999e-06 2024-04-19T02:04:08.752 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.752 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.752 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:08.752 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.775694+0000", 2024-04-19T02:04:08.752 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.2540000000000003e-06 2024-04-19T02:04:08.752 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.752 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.752 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:08.752 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.775706+0000", 2024-04-19T02:04:08.752 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1654000000000001e-05 2024-04-19T02:04:08.752 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.752 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.752 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:08.753 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.775715+0000", 2024-04-19T02:04:08.753 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0189999999999995e-06 2024-04-19T02:04:08.753 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.753 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.753 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:08.753 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.775831+0000", 2024-04-19T02:04:08.753 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000115606 2024-04-19T02:04:08.754 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.754 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.754 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:08.754 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.776060+0000", 2024-04-19T02:04:08.754 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00022901700000000001 2024-04-19T02:04:08.754 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.754 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.754 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:08.754 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.786166+0000", 2024-04-19T02:04:08.754 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010106636 2024-04-19T02:04:08.754 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.754 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:08.754 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.755 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.755 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.755 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s7 46/26 ECSubWrite(tid=58, reqid=osd.1.0:58, at_version=46'44, trim_to=0'0, roll_forward_to=45'42))", 2024-04-19T02:04:08.755 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.059818+0000", 2024-04-19T02:04:08.755 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 38.690635778000001, 2024-04-19T02:04:08.755 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012903721, 2024-04-19T02:04:08.755 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:08.755 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:08.755 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:08.755 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:08.755 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.755 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:08.755 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.059818+0000", 2024-04-19T02:04:08.755 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.755 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.756 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.756 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:08.756 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.059818+0000", 2024-04-19T02:04:08.756 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.756 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.756 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.756 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:08.756 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.059818+0000", 2024-04-19T02:04:08.756 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4900000000000002e-07 2024-04-19T02:04:08.756 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.756 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.756 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:08.756 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.059841+0000", 2024-04-19T02:04:08.756 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3317999999999999e-05 2024-04-19T02:04:08.757 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.757 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.757 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:08.757 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.066536+0000", 2024-04-19T02:04:08.757 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0066946920000000004 2024-04-19T02:04:08.757 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.757 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.757 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:08.757 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.066552+0000", 2024-04-19T02:04:08.757 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5758999999999999e-05 2024-04-19T02:04:08.757 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.757 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.757 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:08.757 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.066653+0000", 2024-04-19T02:04:08.757 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000101648 2024-04-19T02:04:08.758 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.758 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.758 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:08.758 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.066667+0000", 2024-04-19T02:04:08.758 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3648999999999999e-05 2024-04-19T02:04:08.758 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.758 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.758 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:08.758 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.072672+0000", 2024-04-19T02:04:08.758 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0060046709999999996 2024-04-19T02:04:08.758 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.758 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.758 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:08.758 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.072721+0000", 2024-04-19T02:04:08.759 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9734999999999997e-05 2024-04-19T02:04:08.759 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.759 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:08.759 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.759 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.759 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.759 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:415 3.1s0 3:874b3480:::smithi05930959-36:head [append 0~720896] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:08.759 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.072647+0000", 2024-04-19T02:04:08.759 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 38.677806719000003, 2024-04-19T02:04:08.759 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.020543788, 2024-04-19T02:04:08.759 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:08.759 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:08.759 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:08.760 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:08.760 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:08.760 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:08.760 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 415 2024-04-19T02:04:08.760 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.760 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:08.760 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.760 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:08.760 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.072647+0000", 2024-04-19T02:04:08.760 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.760 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.760 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.760 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:08.760 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.072647+0000", 2024-04-19T02:04:08.761 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.761 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.761 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.761 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:08.761 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.072649+0000", 2024-04-19T02:04:08.761 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3649999999999998e-06 2024-04-19T02:04:08.761 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.761 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.761 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:08.761 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.073285+0000", 2024-04-19T02:04:08.761 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00063577000000000002 2024-04-19T02:04:08.761 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.761 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.761 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:08.761 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.073308+0000", 2024-04-19T02:04:08.761 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2612e-05 2024-04-19T02:04:08.762 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.762 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.762 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:08.762 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.073319+0000", 2024-04-19T02:04:08.762 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1992e-05 2024-04-19T02:04:08.762 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.762 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.762 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:08.762 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.073482+0000", 2024-04-19T02:04:08.762 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00016297800000000001 2024-04-19T02:04:08.762 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.762 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.762 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:08.763 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.073685+0000", 2024-04-19T02:04:08.763 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00020296299999999999 2024-04-19T02:04:08.763 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.763 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.763 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:08.763 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.076845+0000", 2024-04-19T02:04:08.763 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0031591380000000001 2024-04-19T02:04:08.763 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.763 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.763 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:08.763 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.085769+0000", 2024-04-19T02:04:08.763 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0089239320000000007 2024-04-19T02:04:08.763 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.763 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.764 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:08.764 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.093097+0000", 2024-04-19T02:04:08.764 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0073282130000000001 2024-04-19T02:04:08.764 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.764 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.764 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:08.764 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.093191+0000", 2024-04-19T02:04:08.764 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3825000000000004e-05 2024-04-19T02:04:08.764 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.764 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:08.764 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.764 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.764 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.764 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:416 3.1s0 3:874b3480:::smithi05930959-36:head [append 0~442368] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:08.765 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.077896+0000", 2024-04-19T02:04:08.765 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 38.672557560000001, 2024-04-19T02:04:08.765 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.018242536, 2024-04-19T02:04:08.765 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:08.765 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:08.765 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:08.765 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:08.765 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:08.765 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:08.765 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 416 2024-04-19T02:04:08.765 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.765 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:08.765 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.765 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:08.766 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077896+0000", 2024-04-19T02:04:08.766 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.766 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.766 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.766 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:08.766 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077896+0000", 2024-04-19T02:04:08.766 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.766 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.766 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.766 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:08.766 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077898+0000", 2024-04-19T02:04:08.766 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1959999999999998e-06 2024-04-19T02:04:08.766 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.766 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.767 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:08.767 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.078421+0000", 2024-04-19T02:04:08.767 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00052314500000000003 2024-04-19T02:04:08.767 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.767 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.767 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:08.767 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.078441+0000", 2024-04-19T02:04:08.767 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9763999999999999e-05 2024-04-19T02:04:08.767 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.767 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.767 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:08.767 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.078453+0000", 2024-04-19T02:04:08.767 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1724e-05 2024-04-19T02:04:08.767 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.767 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.767 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:08.768 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.079580+0000", 2024-04-19T02:04:08.768 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0011274550000000001 2024-04-19T02:04:08.768 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.768 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.768 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:08.768 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.079689+0000", 2024-04-19T02:04:08.768 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108942 2024-04-19T02:04:08.768 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.768 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.768 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:08.768 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080934+0000", 2024-04-19T02:04:08.768 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001245187 2024-04-19T02:04:08.768 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.768 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.768 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:08.769 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.085771+0000", 2024-04-19T02:04:08.769 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.004836707 2024-04-19T02:04:08.769 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.769 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.769 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:08.769 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.096060+0000", 2024-04-19T02:04:08.769 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010288976 2024-04-19T02:04:08.769 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.769 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.769 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:08.769 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.096138+0000", 2024-04-19T02:04:08.769 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.8440000000000001e-05 2024-04-19T02:04:08.769 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.769 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:08.769 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.769 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.770 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.770 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:417 3.1s0 3:874b3480:::smithi05930959-36:head [append 0~327680] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:08.770 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.079192+0000", 2024-04-19T02:04:08.770 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 38.671261029999997, 2024-04-19T02:04:08.770 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.01712928, 2024-04-19T02:04:08.770 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:08.770 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:08.770 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:08.770 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:08.770 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:08.770 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:08.770 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 417 2024-04-19T02:04:08.770 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.770 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:08.770 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.771 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:08.771 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.079192+0000", 2024-04-19T02:04:08.771 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.771 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.771 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.771 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:08.771 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.079192+0000", 2024-04-19T02:04:08.771 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.771 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.771 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.771 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:08.771 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.079195+0000", 2024-04-19T02:04:08.771 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1880000000000001e-06 2024-04-19T02:04:08.771 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.772 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.772 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:08.772 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.079506+0000", 2024-04-19T02:04:08.772 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00031092000000000001 2024-04-19T02:04:08.772 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.772 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.772 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:08.772 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.079526+0000", 2024-04-19T02:04:08.772 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0404e-05 2024-04-19T02:04:08.772 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.772 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.772 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:08.772 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.079540+0000", 2024-04-19T02:04:08.772 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4233000000000001e-05 2024-04-19T02:04:08.772 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.772 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.772 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:08.773 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.081867+0000", 2024-04-19T02:04:08.773 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0023265030000000002 2024-04-19T02:04:08.773 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.773 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.773 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:08.773 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.082009+0000", 2024-04-19T02:04:08.773 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014187299999999999 2024-04-19T02:04:08.773 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.773 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.773 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:08.773 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.083078+0000", 2024-04-19T02:04:08.773 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0010689499999999999 2024-04-19T02:04:08.773 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.773 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.773 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:08.774 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.085773+0000", 2024-04-19T02:04:08.774 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002695771 2024-04-19T02:04:08.774 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.774 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.774 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:08.774 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.096256+0000", 2024-04-19T02:04:08.774 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010482551 2024-04-19T02:04:08.774 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.774 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.774 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:08.774 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.096322+0000", 2024-04-19T02:04:08.774 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.5887000000000001e-05 2024-04-19T02:04:08.774 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.774 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:08.774 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.775 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.775 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.775 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:418 3.1s0 3:874b3480:::smithi05930959-36:head [setxattr _header (56) in=7b] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:08.775 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.080040+0000", 2024-04-19T02:04:08.775 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 38.670413029999999, 2024-04-19T02:04:08.775 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.016869248, 2024-04-19T02:04:08.775 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:08.775 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:08.775 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:08.775 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:08.775 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:08.775 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:08.775 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 418 2024-04-19T02:04:08.775 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.776 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:08.776 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.776 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:08.776 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080040+0000", 2024-04-19T02:04:08.776 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.776 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.776 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.776 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:08.776 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080040+0000", 2024-04-19T02:04:08.776 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.776 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.776 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.776 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:08.776 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080042+0000", 2024-04-19T02:04:08.776 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.215e-06 2024-04-19T02:04:08.776 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.776 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.777 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:08.777 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080049+0000", 2024-04-19T02:04:08.777 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.3180000000000001e-06 2024-04-19T02:04:08.777 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.777 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.777 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:08.777 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080067+0000", 2024-04-19T02:04:08.777 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7805999999999999e-05 2024-04-19T02:04:08.777 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.777 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.777 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:08.778 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080078+0000", 2024-04-19T02:04:08.778 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1262e-05 2024-04-19T02:04:08.778 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.778 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.778 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:08.778 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.084057+0000", 2024-04-19T02:04:08.778 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0039794039999999998 2024-04-19T02:04:08.778 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.778 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.778 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:08.778 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.084161+0000", 2024-04-19T02:04:08.778 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000103862 2024-04-19T02:04:08.779 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.779 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.779 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:08.779 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.084626+0000", 2024-04-19T02:04:08.779 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00046457200000000001 2024-04-19T02:04:08.779 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.779 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.779 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:08.779 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.086690+0000", 2024-04-19T02:04:08.779 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0020638119999999999 2024-04-19T02:04:08.779 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.779 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.779 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:08.780 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.096836+0000", 2024-04-19T02:04:08.780 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.01014584 2024-04-19T02:04:08.780 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.780 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.780 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:08.780 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.096910+0000", 2024-04-19T02:04:08.780 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.4157000000000002e-05 2024-04-19T02:04:08.780 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.780 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:08.780 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.780 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.780 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.780 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:419 3.1s0 3:874b3480:::smithi05930959-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:08.780 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.080096+0000", 2024-04-19T02:04:08.781 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 38.670357805999998, 2024-04-19T02:04:08.781 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.019302687999999998, 2024-04-19T02:04:08.781 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:08.781 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:08.781 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:04:08.781 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:08.781 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:08.781 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:08.781 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 419 2024-04-19T02:04:08.781 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.781 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:08.781 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.781 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:08.781 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080096+0000", 2024-04-19T02:04:08.781 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.782 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.782 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.782 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:08.782 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080096+0000", 2024-04-19T02:04:08.782 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.782 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.782 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.782 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:08.782 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080096+0000", 2024-04-19T02:04:08.782 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.1200000000000002e-07 2024-04-19T02:04:08.782 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.782 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.782 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:08.782 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080100+0000", 2024-04-19T02:04:08.783 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.506e-06 2024-04-19T02:04:08.783 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.783 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.783 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:08.783 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080111+0000", 2024-04-19T02:04:08.783 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0879e-05 2024-04-19T02:04:08.783 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.783 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.783 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:08.783 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.080119+0000", 2024-04-19T02:04:08.783 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.9629999999999998e-06 2024-04-19T02:04:08.783 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.783 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.783 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:08.783 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.085126+0000", 2024-04-19T02:04:08.784 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0050074280000000004 2024-04-19T02:04:08.784 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.784 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.784 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-04-19T02:04:08.784 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.085230+0000", 2024-04-19T02:04:08.784 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000103589 2024-04-19T02:04:08.784 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.784 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.784 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:08.784 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.097332+0000", 2024-04-19T02:04:08.784 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012101796999999999 2024-04-19T02:04:08.784 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.785 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.785 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:08.785 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.097477+0000", 2024-04-19T02:04:08.785 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014539899999999999 2024-04-19T02:04:08.785 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.785 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.785 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:08.785 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.099398+0000", 2024-04-19T02:04:08.785 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0019214150000000001 2024-04-19T02:04:08.785 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.785 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:08.785 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.786 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.786 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.786 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(osd.3.18:3 3.1s0 3:874b3480:::smithi05930959-36:head [assert-version v11,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:08.786 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.195845+0000", 2024-04-19T02:04:08.786 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 38.554608166999998, 2024-04-19T02:04:08.786 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012426504, 2024-04-19T02:04:08.786 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:08.786 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:08.786 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:04:08.786 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:08.786 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.786 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:08.786 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195845+0000", 2024-04-19T02:04:08.787 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.787 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.787 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.787 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:08.787 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195845+0000", 2024-04-19T02:04:08.787 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.787 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.787 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.787 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:08.787 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195847+0000", 2024-04-19T02:04:08.787 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0090000000000002e-06 2024-04-19T02:04:08.787 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.787 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.787 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:08.788 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195856+0000", 2024-04-19T02:04:08.788 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5650000000000002e-06 2024-04-19T02:04:08.788 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.788 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.788 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:08.788 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195871+0000", 2024-04-19T02:04:08.788 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5072000000000001e-05 2024-04-19T02:04:08.788 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.788 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.788 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:08.788 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195879+0000", 2024-04-19T02:04:08.788 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3680000000000002e-06 2024-04-19T02:04:08.788 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.789 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.789 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:08.789 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195978+0000", 2024-04-19T02:04:08.789 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.8202000000000006e-05 2024-04-19T02:04:08.789 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.789 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.789 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:08.789 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.196085+0000", 2024-04-19T02:04:08.789 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000107506 2024-04-19T02:04:08.789 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.789 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.789 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:08.789 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.208272+0000", 2024-04-19T02:04:08.789 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012186782 2024-04-19T02:04:08.789 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.790 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:08.790 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.790 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.790 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.790 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:465 3.1s0 3:874b3480:::smithi05930959-36:head [append 0~786432] snapc a=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:04:08.790 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:31.156851+0000", 2024-04-19T02:04:08.790 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 37.593602228999998, 2024-04-19T02:04:08.790 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0094295679999999993, 2024-04-19T02:04:08.790 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:08.790 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:08.790 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:08.790 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:08.791 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:08.791 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:08.791 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 465 2024-04-19T02:04:08.791 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.791 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:08.791 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.791 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:08.791 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.156851+0000", 2024-04-19T02:04:08.791 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.791 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.791 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.791 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:08.792 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.156851+0000", 2024-04-19T02:04:08.792 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.792 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.792 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.792 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:08.792 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.156854+0000", 2024-04-19T02:04:08.792 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8030000000000001e-06 2024-04-19T02:04:08.792 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.792 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.792 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:08.792 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.157331+0000", 2024-04-19T02:04:08.792 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00047709100000000002 2024-04-19T02:04:08.792 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.792 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.792 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:08.792 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.157357+0000", 2024-04-19T02:04:08.793 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6092e-05 2024-04-19T02:04:08.793 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.793 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.793 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:08.793 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.157371+0000", 2024-04-19T02:04:08.793 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.403e-05 2024-04-19T02:04:08.793 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.793 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.793 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:08.793 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.157471+0000", 2024-04-19T02:04:08.793 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000100049 2024-04-19T02:04:08.793 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.793 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.794 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:08.794 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.157607+0000", 2024-04-19T02:04:08.794 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013581500000000001 2024-04-19T02:04:08.794 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.794 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.794 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:08.794 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.159060+0000", 2024-04-19T02:04:08.794 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001452971 2024-04-19T02:04:08.794 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.794 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.794 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:08.794 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.160520+0000", 2024-04-19T02:04:08.794 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0014597449999999999 2024-04-19T02:04:08.795 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.795 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.795 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:08.795 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.166184+0000", 2024-04-19T02:04:08.795 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0056638799999999996 2024-04-19T02:04:08.795 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.795 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.795 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:08.795 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.166281+0000", 2024-04-19T02:04:08.795 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.7091999999999998e-05 2024-04-19T02:04:08.795 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.795 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:08.795 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.795 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.795 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.795 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:469 3.1s0 3:874b3480:::smithi05930959-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-19T02:04:08.795 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:31.162590+0000", 2024-04-19T02:04:08.796 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 37.587863374000001, 2024-04-19T02:04:08.796 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0086854280000000002, 2024-04-19T02:04:08.796 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:08.796 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:08.796 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:04:08.796 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:08.796 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:08.796 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:08.796 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 469 2024-04-19T02:04:08.796 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.796 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:08.796 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.796 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:08.796 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162590+0000", 2024-04-19T02:04:08.796 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.796 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.797 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.797 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:08.797 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162590+0000", 2024-04-19T02:04:08.797 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.797 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.797 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.797 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:08.797 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162591+0000", 2024-04-19T02:04:08.797 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.1600000000000001e-07 2024-04-19T02:04:08.797 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.797 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.797 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:08.797 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162593+0000", 2024-04-19T02:04:08.797 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.458e-06 2024-04-19T02:04:08.797 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.798 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.798 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:08.798 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162604+0000", 2024-04-19T02:04:08.798 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0869e-05 2024-04-19T02:04:08.798 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.798 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.798 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:08.798 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.162610+0000", 2024-04-19T02:04:08.798 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.2020000000000001e-06 2024-04-19T02:04:08.798 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.798 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.798 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:08.798 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.165692+0000", 2024-04-19T02:04:08.799 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0030820650000000002 2024-04-19T02:04:08.799 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.799 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.799 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-04-19T02:04:08.799 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.165778+0000", 2024-04-19T02:04:08.799 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5927000000000006e-05 2024-04-19T02:04:08.799 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.799 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.799 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:08.799 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.169491+0000", 2024-04-19T02:04:08.799 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0037127610000000002 2024-04-19T02:04:08.799 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.799 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.799 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:08.800 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.169588+0000", 2024-04-19T02:04:08.800 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.6850999999999998e-05 2024-04-19T02:04:08.800 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.800 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.800 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:08.800 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.171276+0000", 2024-04-19T02:04:08.800 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0016877789999999999 2024-04-19T02:04:08.800 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.800 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:08.800 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.800 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.800 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.800 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:646 3.1s0 3:8775257e:::smithi05930959-28:head [read 0~458752,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-19T02:04:08.800 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:46.185680+0000", 2024-04-19T02:04:08.800 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 22.564773203000001, 2024-04-19T02:04:08.801 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0096160059999999999, 2024-04-19T02:04:08.801 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:08.801 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:08.801 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:04:08.801 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:08.801 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:08.801 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:08.801 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 646 2024-04-19T02:04:08.801 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.801 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:08.801 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.801 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:08.801 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185680+0000", 2024-04-19T02:04:08.801 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.802 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.802 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.802 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:08.802 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185680+0000", 2024-04-19T02:04:08.802 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.802 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.802 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.802 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:08.802 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185684+0000", 2024-04-19T02:04:08.802 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0799999999999999e-06 2024-04-19T02:04:08.802 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.802 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.802 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:08.802 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185707+0000", 2024-04-19T02:04:08.803 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2192e-05 2024-04-19T02:04:08.803 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.803 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.803 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:08.803 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185751+0000", 2024-04-19T02:04:08.803 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4098999999999999e-05 2024-04-19T02:04:08.803 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.803 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.803 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:08.803 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185770+0000", 2024-04-19T02:04:08.803 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9116000000000001e-05 2024-04-19T02:04:08.803 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.803 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.803 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:08.804 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185952+0000", 2024-04-19T02:04:08.804 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000181781 2024-04-19T02:04:08.804 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.804 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.804 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "op must wait for map", 2024-04-19T02:04:08.804 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185997+0000", 2024-04-19T02:04:08.804 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5802e-05 2024-04-19T02:04:08.804 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.804 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.804 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:08.804 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.189566+0000", 2024-04-19T02:04:08.804 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.003568323 2024-04-19T02:04:08.804 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.804 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.804 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:08.804 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.189763+0000", 2024-04-19T02:04:08.804 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00019720100000000001 2024-04-19T02:04:08.805 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.805 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.805 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:08.805 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.195296+0000", 2024-04-19T02:04:08.805 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0055334119999999997 2024-04-19T02:04:08.805 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.805 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:08.805 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.805 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.805 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.805 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:647 3.1s0 3:8775257e:::smithi05930959-28:head [read 0~458752,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-19T02:04:08.805 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:46.185813+0000", 2024-04-19T02:04:08.806 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 22.564640310000001, 2024-04-19T02:04:08.806 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010132234, 2024-04-19T02:04:08.806 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:08.806 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:08.806 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:04:08.806 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:08.806 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:08.806 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:08.806 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 647 2024-04-19T02:04:08.806 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.806 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:08.806 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.806 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:08.806 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185813+0000", 2024-04-19T02:04:08.807 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.807 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.807 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.807 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:08.807 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185813+0000", 2024-04-19T02:04:08.807 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.807 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.807 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.807 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:08.807 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185815+0000", 2024-04-19T02:04:08.807 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4699999999999999e-06 2024-04-19T02:04:08.807 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.807 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.807 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:08.807 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185834+0000", 2024-04-19T02:04:08.807 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9494999999999999e-05 2024-04-19T02:04:08.808 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.808 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.808 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:08.808 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185855+0000", 2024-04-19T02:04:08.808 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1265999999999999e-05 2024-04-19T02:04:08.808 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.808 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.808 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:08.808 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185875+0000", 2024-04-19T02:04:08.808 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9772999999999999e-05 2024-04-19T02:04:08.808 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.808 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.808 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:08.808 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.186193+0000", 2024-04-19T02:04:08.809 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00031767699999999999 2024-04-19T02:04:08.809 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.809 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.809 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:04:08.809 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.186210+0000", 2024-04-19T02:04:08.809 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7306999999999999e-05 2024-04-19T02:04:08.809 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.809 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.809 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:08.809 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.191548+0000", 2024-04-19T02:04:08.809 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0053383149999999997 2024-04-19T02:04:08.809 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.809 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.809 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:08.810 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.191764+0000", 2024-04-19T02:04:08.810 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000215109 2024-04-19T02:04:08.810 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.810 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.810 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:08.810 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.195945+0000", 2024-04-19T02:04:08.810 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.004181822 2024-04-19T02:04:08.810 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.810 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:08.810 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.810 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.811 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.811 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:648 3.1s0 3:8775257e:::smithi05930959-28:head [read 0~458752,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-04-19T02:04:08.811 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:46.185903+0000", 2024-04-19T02:04:08.811 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 22.564550213, 2024-04-19T02:04:08.811 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.011792667999999999, 2024-04-19T02:04:08.811 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:08.811 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:08.811 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:04:08.811 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:08.811 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:08.811 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:08.811 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 648 2024-04-19T02:04:08.812 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.812 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:08.812 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.812 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:08.812 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185903+0000", 2024-04-19T02:04:08.812 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.812 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.812 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.812 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:08.812 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185903+0000", 2024-04-19T02:04:08.812 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.812 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.813 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.813 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:08.813 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185905+0000", 2024-04-19T02:04:08.813 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.246e-06 2024-04-19T02:04:08.813 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.813 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.813 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:08.813 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185913+0000", 2024-04-19T02:04:08.813 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5199999999999997e-06 2024-04-19T02:04:08.813 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.813 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.813 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:08.813 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185957+0000", 2024-04-19T02:04:08.813 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4212000000000003e-05 2024-04-19T02:04:08.814 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.814 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.814 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:08.814 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.185983+0000", 2024-04-19T02:04:08.814 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6118999999999998e-05 2024-04-19T02:04:08.814 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.814 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.814 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:08.814 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.186272+0000", 2024-04-19T02:04:08.814 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00028864 2024-04-19T02:04:08.814 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.814 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.814 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:04:08.814 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.186285+0000", 2024-04-19T02:04:08.815 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3446e-05 2024-04-19T02:04:08.815 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.815 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.815 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:08.815 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.193694+0000", 2024-04-19T02:04:08.815 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0074089300000000002 2024-04-19T02:04:08.815 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.815 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.815 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:08.815 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.193872+0000", 2024-04-19T02:04:08.815 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000177307 2024-04-19T02:04:08.815 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.816 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.816 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:08.816 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.197696+0000", 2024-04-19T02:04:08.816 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0038242480000000001 2024-04-19T02:04:08.816 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.816 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:08.816 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.816 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.816 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.816 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:800 3.1s0 3:8775257e:::smithi05930959-28:head [append 0~425984] snapc 19=[19,17,16,15,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-19T02:04:08.816 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:54.081264+0000", 2024-04-19T02:04:08.816 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 14.669189758, 2024-04-19T02:04:08.817 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010268286999999999, 2024-04-19T02:04:08.817 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:08.817 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:08.817 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:08.817 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:08.817 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:08.817 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:08.817 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 800 2024-04-19T02:04:08.817 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.817 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:08.817 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.817 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:08.817 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.081264+0000", 2024-04-19T02:04:08.818 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.818 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.818 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.818 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:08.818 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.081264+0000", 2024-04-19T02:04:08.818 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.818 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.818 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.818 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:08.818 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.081266+0000", 2024-04-19T02:04:08.818 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0209999999999998e-06 2024-04-19T02:04:08.818 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.818 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.818 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:08.819 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.081690+0000", 2024-04-19T02:04:08.819 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00042431700000000001 2024-04-19T02:04:08.819 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.819 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.819 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:08.819 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.081710+0000", 2024-04-19T02:04:08.819 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9517999999999998e-05 2024-04-19T02:04:08.819 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.819 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.819 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:08.819 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.081719+0000", 2024-04-19T02:04:08.819 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.9239999999999996e-06 2024-04-19T02:04:08.819 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.820 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.820 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:08.820 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.081971+0000", 2024-04-19T02:04:08.820 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000252038 2024-04-19T02:04:08.820 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.820 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.820 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:08.820 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.082076+0000", 2024-04-19T02:04:08.820 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000105772 2024-04-19T02:04:08.820 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.820 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.820 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:08.820 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.083554+0000", 2024-04-19T02:04:08.821 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0014774899999999999 2024-04-19T02:04:08.821 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.821 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.821 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:08.821 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.085032+0000", 2024-04-19T02:04:08.821 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001478112 2024-04-19T02:04:08.821 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.821 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.821 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:08.821 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.091414+0000", 2024-04-19T02:04:08.821 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0063820450000000003 2024-04-19T02:04:08.821 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.821 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.821 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:08.821 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.091532+0000", 2024-04-19T02:04:08.821 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00011805 2024-04-19T02:04:08.822 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.822 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:08.822 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.822 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.822 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.822 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:801 3.1s0 3:8775257e:::smithi05930959-28:head [append 0~573440] snapc 19=[19,17,16,15,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-19T02:04:08.822 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:54.085113+0000", 2024-04-19T02:04:08.822 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 14.665340283000001, 2024-04-19T02:04:08.822 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0095650130000000007, 2024-04-19T02:04:08.822 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:08.822 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:08.822 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:08.823 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:08.823 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:08.823 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:08.823 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 801 2024-04-19T02:04:08.823 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.823 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:08.823 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.823 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:08.823 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.085113+0000", 2024-04-19T02:04:08.823 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.823 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.823 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.823 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:08.823 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.085113+0000", 2024-04-19T02:04:08.824 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.824 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.824 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.824 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:08.824 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.085115+0000", 2024-04-19T02:04:08.824 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.809e-06 2024-04-19T02:04:08.824 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.824 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.824 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:08.824 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.085534+0000", 2024-04-19T02:04:08.824 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00041926499999999999 2024-04-19T02:04:08.824 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.824 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.825 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:08.825 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.085555+0000", 2024-04-19T02:04:08.825 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1072000000000001e-05 2024-04-19T02:04:08.825 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.825 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.825 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:08.825 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.085564+0000", 2024-04-19T02:04:08.825 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.9749999999999996e-06 2024-04-19T02:04:08.825 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.825 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.825 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:08.825 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.085665+0000", 2024-04-19T02:04:08.825 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010073699999999999 2024-04-19T02:04:08.825 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.826 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.826 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:08.826 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.085781+0000", 2024-04-19T02:04:08.826 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000116165 2024-04-19T02:04:08.826 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.826 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.826 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:08.826 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.087025+0000", 2024-04-19T02:04:08.826 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0012436700000000001 2024-04-19T02:04:08.826 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.826 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.826 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:08.826 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.091275+0000", 2024-04-19T02:04:08.826 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0042499870000000002 2024-04-19T02:04:08.827 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.827 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.827 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:08.827 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.094448+0000", 2024-04-19T02:04:08.827 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0031731770000000001 2024-04-19T02:04:08.827 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.827 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.827 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:08.827 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.094678+0000", 2024-04-19T02:04:08.827 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000230156 2024-04-19T02:04:08.827 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.827 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:08.827 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.827 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.827 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.828 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:802 3.1s0 3:8775257e:::smithi05930959-28:head [append 0~425984] snapc 19=[19,17,16,15,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-19T02:04:08.828 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:54.088456+0000", 2024-04-19T02:04:08.828 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 14.661997244, 2024-04-19T02:04:08.828 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0091163419999999995, 2024-04-19T02:04:08.828 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:08.828 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:08.828 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:08.828 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:08.828 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:08.828 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:08.828 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 802 2024-04-19T02:04:08.828 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.829 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:08.829 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.829 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:08.829 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.088456+0000", 2024-04-19T02:04:08.829 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.829 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.829 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.829 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:08.829 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.088456+0000", 2024-04-19T02:04:08.829 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.829 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.829 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.829 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:08.829 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.088458+0000", 2024-04-19T02:04:08.830 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0370000000000001e-06 2024-04-19T02:04:08.830 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.830 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.830 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:08.830 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.088890+0000", 2024-04-19T02:04:08.830 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00043142300000000001 2024-04-19T02:04:08.830 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.830 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.830 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:08.830 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.088906+0000", 2024-04-19T02:04:08.830 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6191000000000001e-05 2024-04-19T02:04:08.830 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.830 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.830 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:08.831 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.088916+0000", 2024-04-19T02:04:08.831 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9990000000000003e-06 2024-04-19T02:04:08.831 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.831 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.831 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:08.831 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.089000+0000", 2024-04-19T02:04:08.831 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3682000000000003e-05 2024-04-19T02:04:08.831 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.831 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.831 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:08.831 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.089143+0000", 2024-04-19T02:04:08.831 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014330300000000001 2024-04-19T02:04:08.831 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.832 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.832 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:08.832 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.090386+0000", 2024-04-19T02:04:08.832 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001242997 2024-04-19T02:04:08.832 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.832 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.832 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:08.832 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.091713+0000", 2024-04-19T02:04:08.832 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001327286 2024-04-19T02:04:08.832 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.832 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.832 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:08.832 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.097489+0000", 2024-04-19T02:04:08.833 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0057758439999999996 2024-04-19T02:04:08.833 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.833 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.833 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:08.833 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.097573+0000", 2024-04-19T02:04:08.833 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3579999999999996e-05 2024-04-19T02:04:08.833 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.833 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:08.833 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.833 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.833 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.833 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:804 3.1s0 3:8775257e:::smithi05930959-28:head [append 0~425984] snapc 19=[19,17,16,15,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-19T02:04:08.834 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:54.094511+0000", 2024-04-19T02:04:08.834 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 14.655942485000001, 2024-04-19T02:04:08.834 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.011389521, 2024-04-19T02:04:08.834 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:08.834 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:08.834 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:08.834 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:08.834 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:08.834 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:08.835 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 804 2024-04-19T02:04:08.835 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.835 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:08.835 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.835 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:08.835 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.094511+0000", 2024-04-19T02:04:08.835 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.835 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.835 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.835 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:08.835 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.094511+0000", 2024-04-19T02:04:08.835 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.835 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.836 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.836 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:08.836 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.094513+0000", 2024-04-19T02:04:08.836 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6610000000000001e-06 2024-04-19T02:04:08.836 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.836 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.836 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:08.836 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.094702+0000", 2024-04-19T02:04:08.836 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00018909100000000001 2024-04-19T02:04:08.836 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.836 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.836 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:08.836 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.094715+0000", 2024-04-19T02:04:08.837 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3582e-05 2024-04-19T02:04:08.837 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.837 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.837 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:08.837 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.094726+0000", 2024-04-19T02:04:08.837 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0876000000000001e-05 2024-04-19T02:04:08.837 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.837 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.837 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:08.837 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.094787+0000", 2024-04-19T02:04:08.837 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0891999999999999e-05 2024-04-19T02:04:08.837 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.837 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.837 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:08.838 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.094902+0000", 2024-04-19T02:04:08.838 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00011495899999999999 2024-04-19T02:04:08.838 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.838 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.838 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:08.838 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.095884+0000", 2024-04-19T02:04:08.838 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00098190899999999999 2024-04-19T02:04:08.838 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.838 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.838 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:08.838 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.097738+0000", 2024-04-19T02:04:08.838 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0018543419999999999 2024-04-19T02:04:08.839 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.839 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.839 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:08.839 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.105836+0000", 2024-04-19T02:04:08.839 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.008097946 2024-04-19T02:04:08.839 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.839 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.839 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:08.839 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.105901+0000", 2024-04-19T02:04:08.839 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.4263000000000005e-05 2024-04-19T02:04:08.839 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.839 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:08.839 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.840 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.840 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.840 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:805 3.1s0 3:8775257e:::smithi05930959-28:head [setxattr _header (57) in=7b] snapc 19=[19,17,16,15,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-19T02:04:08.840 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:54.094747+0000", 2024-04-19T02:04:08.840 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 14.655706435999999, 2024-04-19T02:04:08.840 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.011396236000000001, 2024-04-19T02:04:08.840 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:08.840 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:08.840 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:08.840 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:08.840 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:08.840 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:08.840 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 805 2024-04-19T02:04:08.841 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.841 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:08.841 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.841 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:08.841 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.094747+0000", 2024-04-19T02:04:08.841 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.841 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.841 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.841 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:08.841 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.094747+0000", 2024-04-19T02:04:08.841 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.841 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.841 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.841 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:08.841 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.094748+0000", 2024-04-19T02:04:08.842 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0389999999999999e-06 2024-04-19T02:04:08.842 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.842 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.842 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:08.842 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.094754+0000", 2024-04-19T02:04:08.842 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.4650000000000002e-06 2024-04-19T02:04:08.842 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.842 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.842 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:08.842 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.094763+0000", 2024-04-19T02:04:08.842 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.4730000000000006e-06 2024-04-19T02:04:08.842 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.842 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.842 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:08.842 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.094771+0000", 2024-04-19T02:04:08.843 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.5580000000000003e-06 2024-04-19T02:04:08.843 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.843 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.843 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:08.843 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.096421+0000", 2024-04-19T02:04:08.843 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00165036 2024-04-19T02:04:08.843 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.843 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.843 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:08.843 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.096492+0000", 2024-04-19T02:04:08.843 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.1032999999999996e-05 2024-04-19T02:04:08.843 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.843 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.843 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:08.843 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.096885+0000", 2024-04-19T02:04:08.843 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00039302899999999998 2024-04-19T02:04:08.844 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.844 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.844 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:08.844 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.098231+0000", 2024-04-19T02:04:08.844 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0013455349999999999 2024-04-19T02:04:08.844 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.844 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.844 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:08.844 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.106037+0000", 2024-04-19T02:04:08.844 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0078061889999999998 2024-04-19T02:04:08.844 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.844 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.844 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:08.844 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.106143+0000", 2024-04-19T02:04:08.844 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000106555 2024-04-19T02:04:08.845 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.845 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:08.845 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.845 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.845 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.845 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:806 3.1s0 3:8775257e:::smithi05930959-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e70)", 2024-04-19T02:04:08.845 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:54.094785+0000", 2024-04-19T02:04:08.845 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 14.655667997, 2024-04-19T02:04:08.845 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.016130288999999999, 2024-04-19T02:04:08.845 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:08.845 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:08.845 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:04:08.845 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:08.845 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:08.845 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:08.846 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 806 2024-04-19T02:04:08.846 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.846 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:08.846 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.846 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:08.846 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.094785+0000", 2024-04-19T02:04:08.846 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.846 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.846 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.846 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:08.846 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.094785+0000", 2024-04-19T02:04:08.846 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.846 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.847 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.847 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:08.847 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.094786+0000", 2024-04-19T02:04:08.847 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.6599999999999994e-07 2024-04-19T02:04:08.847 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.847 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.847 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:08.847 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.094790+0000", 2024-04-19T02:04:08.847 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.371e-06 2024-04-19T02:04:08.847 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.847 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.847 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:08.847 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.094800+0000", 2024-04-19T02:04:08.847 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9939999999999995e-06 2024-04-19T02:04:08.847 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.848 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.848 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:08.848 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.094806+0000", 2024-04-19T02:04:08.848 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.4640000000000003e-06 2024-04-19T02:04:08.848 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.848 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.848 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:08.848 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.097674+0000", 2024-04-19T02:04:08.848 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002868069 2024-04-19T02:04:08.848 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.848 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.848 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-04-19T02:04:08.848 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.097732+0000", 2024-04-19T02:04:08.849 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7648000000000001e-05 2024-04-19T02:04:08.849 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.849 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.849 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:08.849 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.106644+0000", 2024-04-19T02:04:08.849 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0089121990000000009 2024-04-19T02:04:08.849 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.849 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.849 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:08.849 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.106827+0000", 2024-04-19T02:04:08.849 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00018297100000000001 2024-04-19T02:04:08.849 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.849 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.849 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:08.850 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.110916+0000", 2024-04-19T02:04:08.850 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0040886070000000002 2024-04-19T02:04:08.850 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.850 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:08.850 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.850 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.850 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.850 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:890 3.1s0 3:874b3480:::smithi05930959-36:head [create,setxattr `\\SSSfaPsYfUrUsdSgQgPSapRRms^pqX[osPqdeo (1122) in=40b,setxattr [ (14664) in=1b,setxattr ]_ZacOjR\\V[PolVkSemZdQgglWneOb`chhho^qsi (1365) in=40b,setxattr dlc^[PchQtttbUqnWU```dtpV[iqhiYUtii\\YjTS (1049) in=40b,setxattr pU[nrRrZQQalsblk`ltfaZqrOram[s]cbdhqjSOr (14664) in=40b,setxattr NsheYe]]`bSVTqp\\eWWl (14664) in=20b,setxattr gscckRPkoUNsYbQjkjm_XOsRZllQlNYf^Q`altka (1894) in=40b,setxattr WsNgSgepYqnlrSUek[c\\mS`YlVcq]aXNdhNZqOU_ (803) in=40b,setxattr r\\eN\\d_t[n`tS (14664) in=13b,setxattr g[QgabsXePRNsPqnNqdiZoTaT_]derTZaajrtZPU (1086) in=40b,setxattr SXZemNXWtQUden_WetNWPtrXbjPZYcPXbbU`YSQb (138) in=40b,setxattr OoOSrPaT_VZXQdOh]`[is_WOWeqkpXa]bi\\epXSP (1226) in=40b,setxattr SiXkffqeiTaTiRYVhZeeSeb\\Qkmm^]UseelOaZ]q (1715) in=40b,setxattr nbg_sNYksteVOZjdrW\\P\\ebkpgXci`^pq_nfcXjr (243) in=40b,setxattr ^WgZ\\tgTZprq`o\\YXUtkYak[tfSW[aotVZesNaNV (735) in=40b,setxattr Zr]jhehfqZbYohtW_kqUhRe[NotmVdcfT[atapt] (1683) in=40b,setxattr ^rYSPceWRnj_nVWePZgm_k]WXkOPTd]Utragh]on (630) in=40b,setxattr _[UVkTkSmaX`mfTX_Qp^k^a\\clWeYeWhNkY_OWn\\ (6736) in=40b,setxattr ]p (14664) in=2b,setxattr ^Rtio`SsUNTPt_daN^XiUZbhTVgfa`_SV`famqjr (1927) in=40b] snapc 1d=[1d,1c,1b,19,17,16,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-04-19T02:04:08.850 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:04:00.157064+0000", 2024-04-19T02:04:08.850 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 8.5933892430000007, 2024-04-19T02:04:08.850 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.020005196999999999, 2024-04-19T02:04:08.850 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:08.850 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:08.850 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:08.851 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:08.851 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:08.851 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:08.851 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 890 2024-04-19T02:04:08.851 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.851 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:08.851 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.851 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:08.851 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.157064+0000", 2024-04-19T02:04:08.851 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.851 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.851 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.851 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:08.852 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.157064+0000", 2024-04-19T02:04:08.852 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:08.852 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.852 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.852 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:08.852 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.157066+0000", 2024-04-19T02:04:08.852 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8139999999999999e-06 2024-04-19T02:04:08.852 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.852 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.852 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:08.852 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.157104+0000", 2024-04-19T02:04:08.852 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.8090999999999998e-05 2024-04-19T02:04:08.852 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.852 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.852 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:08.853 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.157125+0000", 2024-04-19T02:04:08.853 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1265000000000001e-05 2024-04-19T02:04:08.853 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.853 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.853 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:08.853 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.157134+0000", 2024-04-19T02:04:08.853 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.9900000000000003e-06 2024-04-19T02:04:08.853 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.853 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.853 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:08.853 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.157222+0000", 2024-04-19T02:04:08.853 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7905999999999995e-05 2024-04-19T02:04:08.853 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.853 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.854 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:08.854 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.157368+0000", 2024-04-19T02:04:08.854 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014548 2024-04-19T02:04:08.854 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.854 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.854 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:08.854 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.158738+0000", 2024-04-19T02:04:08.854 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001370547 2024-04-19T02:04:08.854 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.854 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.854 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:08.854 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.160692+0000", 2024-04-19T02:04:08.854 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0019535099999999999 2024-04-19T02:04:08.854 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.855 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.855 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:08.855 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.176965+0000", 2024-04-19T02:04:08.855 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.016272701000000001 2024-04-19T02:04:08.855 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:08.855 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:08.855 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:08.855 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.177069+0000", 2024-04-19T02:04:08.855 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000104893 2024-04-19T02:04:08.855 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.855 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:08.855 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.855 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:08.855 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:08.855 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:04:08.857 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-19T02:04:08.859 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:04:08.861+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:04:08.901 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2024-04-19T02:04:08.959 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:04:08.961+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:04:09.007 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:04:09.007 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:04:09.007 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-04-19T02:04:09.007 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:04:09.020 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-19T02:04:09.059 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is set 2024-04-19T02:04:09.060 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:04:09.061+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:04:09.061 INFO:tasks.rados.rados.0.smithi059.stdout:371: done (2 left) 2024-04-19T02:04:09.062 INFO:tasks.rados.rados.0.smithi059.stdout:372: done (1 left) 2024-04-19T02:04:09.062 INFO:tasks.rados.rados.0.smithi059.stdout:373: done (0 left) 2024-04-19T02:04:09.062 INFO:tasks.rados.rados.0.smithi059.stdout:374: snap_remove snap 31 2024-04-19T02:04:09.104 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2024-04-19T02:04:09.143 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:04:09.144 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:04:09.144 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-04-19T02:04:09.144 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-04-19T02:04:09.144 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:04:09.158 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-19T02:04:09.160 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:04:09.161+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:04:09.261 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:04:09.261+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:04:09.302 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 reset_purged_snaps_last 2024-04-19T02:04:09.327 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:04:09.327 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-04-19T02:04:09.327 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-04-19T02:04:09.327 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-04-19T02:04:09.327 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.327 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:344 3.4s0 3:2277bf56:::smithi05930959-44:head [append 0~606208] snapc 8=[8,6,5] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:04:09.328 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.735396+0000", 2024-04-19T02:04:09.328 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 44.593058569, 2024-04-19T02:04:09.328 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.014124197999999999, 2024-04-19T02:04:09.328 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:09.328 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:09.328 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:09.328 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:09.328 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:09.328 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:09.328 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 344 2024-04-19T02:04:09.328 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.328 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:09.328 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.328 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:09.328 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.735396+0000", 2024-04-19T02:04:09.328 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.328 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.329 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.329 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:09.329 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.735396+0000", 2024-04-19T02:04:09.329 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.329 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.329 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.329 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:09.329 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.735399+0000", 2024-04-19T02:04:09.329 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.32e-06 2024-04-19T02:04:09.329 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.329 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.329 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:09.329 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.736033+0000", 2024-04-19T02:04:09.329 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000633689 2024-04-19T02:04:09.329 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.330 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.330 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:09.330 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.736074+0000", 2024-04-19T02:04:09.330 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1020999999999999e-05 2024-04-19T02:04:09.330 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.330 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.330 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:09.330 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.736094+0000", 2024-04-19T02:04:09.330 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0339e-05 2024-04-19T02:04:09.330 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.330 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.330 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.330 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.736202+0000", 2024-04-19T02:04:09.330 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000107899 2024-04-19T02:04:09.331 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.331 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.331 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "op must wait for map", 2024-04-19T02:04:09.331 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.736212+0000", 2024-04-19T02:04:09.331 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.7259999999999997e-06 2024-04-19T02:04:09.331 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.331 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.331 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.331 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.738290+0000", 2024-04-19T02:04:09.331 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002078725 2024-04-19T02:04:09.331 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.331 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.331 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:09.331 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.738359+0000", 2024-04-19T02:04:09.331 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8177999999999997e-05 2024-04-19T02:04:09.332 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.332 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.332 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:09.332 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.739412+0000", 2024-04-19T02:04:09.332 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0010534240000000001 2024-04-19T02:04:09.332 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.332 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.332 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:09.332 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.741318+0000", 2024-04-19T02:04:09.332 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001906151 2024-04-19T02:04:09.332 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.332 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.332 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:09.332 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749459+0000", 2024-04-19T02:04:09.333 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0081410830000000003 2024-04-19T02:04:09.333 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.333 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.333 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:09.333 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749520+0000", 2024-04-19T02:04:09.333 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0643000000000002e-05 2024-04-19T02:04:09.333 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.333 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:09.333 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.333 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.333 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.333 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:348 3.4s0 3:2277bf56:::smithi05930959-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:04:09.333 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.746345+0000", 2024-04-19T02:04:09.333 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 44.582109349, 2024-04-19T02:04:09.333 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012391231000000001, 2024-04-19T02:04:09.334 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:09.334 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:09.334 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:04:09.334 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:09.334 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:09.334 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:09.334 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 348 2024-04-19T02:04:09.334 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.334 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:09.334 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.334 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:09.334 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746345+0000", 2024-04-19T02:04:09.334 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.335 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.335 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.335 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:09.335 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746345+0000", 2024-04-19T02:04:09.335 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.335 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.335 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.335 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:09.335 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746346+0000", 2024-04-19T02:04:09.335 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.5899999999999996e-07 2024-04-19T02:04:09.335 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.335 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.336 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:09.336 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746349+0000", 2024-04-19T02:04:09.336 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2100000000000002e-06 2024-04-19T02:04:09.336 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.336 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.336 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:09.336 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746359+0000", 2024-04-19T02:04:09.336 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.8069999999999994e-06 2024-04-19T02:04:09.336 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.336 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.336 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:09.336 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746366+0000", 2024-04-19T02:04:09.336 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.1969999999999999e-06 2024-04-19T02:04:09.337 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.337 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.337 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.337 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750173+0000", 2024-04-19T02:04:09.337 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0038071400000000001 2024-04-19T02:04:09.337 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.337 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.337 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-04-19T02:04:09.337 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750223+0000", 2024-04-19T02:04:09.337 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9597e-05 2024-04-19T02:04:09.337 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.337 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.337 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.338 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.757010+0000", 2024-04-19T02:04:09.338 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.006787062 2024-04-19T02:04:09.338 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.338 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.338 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:09.338 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.757155+0000", 2024-04-19T02:04:09.338 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000145303 2024-04-19T02:04:09.338 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.338 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.338 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:09.338 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.758736+0000", 2024-04-19T02:04:09.338 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0015812560000000001 2024-04-19T02:04:09.338 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.339 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:09.339 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.339 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.339 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.339 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:374 3.4s0 3:2277bf56:::smithi05930959-44:head [append 0~0] snapc 9=[9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:04:09.339 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:26.758473+0000", 2024-04-19T02:04:09.339 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 42.569981102, 2024-04-19T02:04:09.339 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.032817486999999999, 2024-04-19T02:04:09.339 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:09.339 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:09.339 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:09.339 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:09.339 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:09.339 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:09.340 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 374 2024-04-19T02:04:09.340 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.340 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:09.340 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.340 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:09.340 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758473+0000", 2024-04-19T02:04:09.340 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.340 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.340 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.340 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:09.340 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758473+0000", 2024-04-19T02:04:09.340 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.340 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.341 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.341 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:09.341 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758475+0000", 2024-04-19T02:04:09.341 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.835e-06 2024-04-19T02:04:09.341 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.341 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.341 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:09.341 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758481+0000", 2024-04-19T02:04:09.341 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0669999999999997e-06 2024-04-19T02:04:09.341 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.341 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.341 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:09.341 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758497+0000", 2024-04-19T02:04:09.341 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6195e-05 2024-04-19T02:04:09.342 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.342 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.342 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:09.342 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758506+0000", 2024-04-19T02:04:09.342 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.1730000000000001e-06 2024-04-19T02:04:09.342 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.342 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.342 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.342 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758624+0000", 2024-04-19T02:04:09.342 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000117336 2024-04-19T02:04:09.342 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.342 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.342 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:09.343 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758812+0000", 2024-04-19T02:04:09.343 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000188684 2024-04-19T02:04:09.343 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.343 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.343 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:09.343 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.760058+0000", 2024-04-19T02:04:09.343 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001245812 2024-04-19T02:04:09.343 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.343 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.343 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:09.343 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.762381+0000", 2024-04-19T02:04:09.343 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0023226549999999999 2024-04-19T02:04:09.343 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.343 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.344 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:09.344 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.791180+0000", 2024-04-19T02:04:09.344 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.028798727 2024-04-19T02:04:09.344 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.344 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.344 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:09.344 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.791291+0000", 2024-04-19T02:04:09.344 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00011100299999999999 2024-04-19T02:04:09.344 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.344 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:09.344 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.344 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.344 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.344 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:375 3.4s0 3:2277bf56:::smithi05930959-44:head [rollback 8] snapc 9=[9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:04:09.345 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:26.758531+0000", 2024-04-19T02:04:09.345 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 42.569923228999997, 2024-04-19T02:04:09.345 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.033180481999999997, 2024-04-19T02:04:09.345 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:09.345 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:09.345 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:09.345 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:09.345 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:09.345 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:09.345 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 375 2024-04-19T02:04:09.345 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.345 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:09.345 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.346 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:09.346 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758531+0000", 2024-04-19T02:04:09.346 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.346 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.346 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.346 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:09.346 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758531+0000", 2024-04-19T02:04:09.346 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.346 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.346 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.346 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:09.346 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758532+0000", 2024-04-19T02:04:09.346 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.5700000000000002e-07 2024-04-19T02:04:09.346 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.347 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.347 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:09.347 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758535+0000", 2024-04-19T02:04:09.347 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4520000000000002e-06 2024-04-19T02:04:09.347 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.347 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.347 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:09.347 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758546+0000", 2024-04-19T02:04:09.347 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1195e-05 2024-04-19T02:04:09.347 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.347 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.347 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:09.347 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758554+0000", 2024-04-19T02:04:09.347 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.2579999999999998e-06 2024-04-19T02:04:09.347 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.347 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.348 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.348 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.760997+0000", 2024-04-19T02:04:09.348 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0024429930000000001 2024-04-19T02:04:09.348 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.348 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.348 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:09.348 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.761153+0000", 2024-04-19T02:04:09.348 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00015617100000000001 2024-04-19T02:04:09.348 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.348 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.348 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:09.348 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.761823+0000", 2024-04-19T02:04:09.348 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00067058000000000005 2024-04-19T02:04:09.348 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.348 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.348 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:09.349 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.765008+0000", 2024-04-19T02:04:09.349 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0031845300000000001 2024-04-19T02:04:09.349 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.349 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.349 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:09.349 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.791630+0000", 2024-04-19T02:04:09.349 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.026622456999999999 2024-04-19T02:04:09.349 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.349 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.349 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:09.349 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.791712+0000", 2024-04-19T02:04:09.349 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.1188999999999998e-05 2024-04-19T02:04:09.349 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.349 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:09.349 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.349 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.350 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.350 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:376 3.4s0 3:2277bf56:::smithi05930959-44:head [append 0~0] snapc 9=[9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:04:09.350 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:26.758565+0000", 2024-04-19T02:04:09.350 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 42.569889471000003, 2024-04-19T02:04:09.350 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.033486091000000003, 2024-04-19T02:04:09.350 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:09.350 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:09.350 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:09.350 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:09.350 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:09.350 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:09.350 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 376 2024-04-19T02:04:09.350 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.350 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:09.351 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.351 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:09.351 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758565+0000", 2024-04-19T02:04:09.351 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.351 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.351 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.351 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:09.351 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758565+0000", 2024-04-19T02:04:09.351 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.351 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.351 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.351 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:09.351 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758565+0000", 2024-04-19T02:04:09.351 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9900000000000001e-07 2024-04-19T02:04:09.351 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.351 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.352 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:09.352 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758568+0000", 2024-04-19T02:04:09.352 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0010000000000002e-06 2024-04-19T02:04:09.352 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.352 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.352 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:09.352 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758576+0000", 2024-04-19T02:04:09.352 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.9719999999999992e-06 2024-04-19T02:04:09.352 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.352 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.352 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:09.352 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.758583+0000", 2024-04-19T02:04:09.352 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.2489999999999998e-06 2024-04-19T02:04:09.352 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.352 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.352 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.353 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.763053+0000", 2024-04-19T02:04:09.353 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0044703809999999998 2024-04-19T02:04:09.353 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.353 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.353 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:09.353 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.763266+0000", 2024-04-19T02:04:09.353 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00021301299999999999 2024-04-19T02:04:09.353 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.353 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.353 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:09.353 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.764685+0000", 2024-04-19T02:04:09.353 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0014195379999999999 2024-04-19T02:04:09.353 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.353 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.353 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:09.353 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.765894+0000", 2024-04-19T02:04:09.354 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0012088859999999999 2024-04-19T02:04:09.354 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.354 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.354 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:09.354 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.791938+0000", 2024-04-19T02:04:09.354 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.026043251999999999 2024-04-19T02:04:09.354 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.354 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.354 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:09.354 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.792051+0000", 2024-04-19T02:04:09.354 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0001133 2024-04-19T02:04:09.354 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.354 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:09.355 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.355 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.355 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.355 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:728 3.4s0 3:2277bf56:::smithi05930959-44:head [delete] snapc 16=[16,15,12,10,c] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e66)", 2024-04-19T02:04:09.355 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:50.059596+0000", 2024-04-19T02:04:09.355 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 19.268858542, 2024-04-19T02:04:09.355 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.015090878, 2024-04-19T02:04:09.355 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:09.355 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:09.355 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:04:09.355 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:09.355 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:09.355 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:09.355 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 728 2024-04-19T02:04:09.355 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.356 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:09.356 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.356 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:09.356 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.059596+0000", 2024-04-19T02:04:09.356 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.356 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.356 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.356 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:09.356 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.059596+0000", 2024-04-19T02:04:09.356 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.356 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.356 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.356 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:09.356 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.059598+0000", 2024-04-19T02:04:09.357 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5160000000000001e-06 2024-04-19T02:04:09.357 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.357 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.357 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:09.357 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.059609+0000", 2024-04-19T02:04:09.357 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1051e-05 2024-04-19T02:04:09.357 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.357 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.358 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:09.358 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.059635+0000", 2024-04-19T02:04:09.358 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6185000000000001e-05 2024-04-19T02:04:09.358 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.358 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.358 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:09.358 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.059648+0000", 2024-04-19T02:04:09.358 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2439999999999999e-05 2024-04-19T02:04:09.358 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.358 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.358 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.358 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.059734+0000", 2024-04-19T02:04:09.358 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.6207999999999999e-05 2024-04-19T02:04:09.358 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.359 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.359 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "op must wait for map", 2024-04-19T02:04:09.359 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.059743+0000", 2024-04-19T02:04:09.359 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.2520000000000005e-06 2024-04-19T02:04:09.359 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.359 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.359 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.359 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.061974+0000", 2024-04-19T02:04:09.359 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0022307889999999999 2024-04-19T02:04:09.359 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.359 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.359 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:09.359 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.062111+0000", 2024-04-19T02:04:09.359 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013688799999999999 2024-04-19T02:04:09.359 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.359 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.360 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:09.360 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.074687+0000", 2024-04-19T02:04:09.360 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012575549 2024-04-19T02:04:09.360 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.360 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:09.360 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.360 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.360 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.360 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:737 3.4s0 3:24202781:::smithi05930959-26:head [append 0~0] snapc 16=[16,15,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-19T02:04:09.360 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:50.075201+0000", 2024-04-19T02:04:09.360 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 19.253253719, 2024-04-19T02:04:09.360 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.014835278, 2024-04-19T02:04:09.360 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:09.360 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:09.360 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:09.361 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:09.361 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:09.361 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:09.361 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 737 2024-04-19T02:04:09.361 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.361 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:09.361 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.361 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:09.361 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.075201+0000", 2024-04-19T02:04:09.361 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.361 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.361 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.361 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:09.361 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.075201+0000", 2024-04-19T02:04:09.361 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.362 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.362 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.362 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:09.362 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.075201+0000", 2024-04-19T02:04:09.362 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.4300000000000003e-07 2024-04-19T02:04:09.362 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.362 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.362 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:09.362 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.075206+0000", 2024-04-19T02:04:09.362 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4669999999999998e-06 2024-04-19T02:04:09.362 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.362 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.362 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:09.362 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.075214+0000", 2024-04-19T02:04:09.363 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2949999999999994e-06 2024-04-19T02:04:09.363 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.363 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.363 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:09.363 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.075220+0000", 2024-04-19T02:04:09.363 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.4169999999999997e-06 2024-04-19T02:04:09.363 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.363 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.363 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.363 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.078117+0000", 2024-04-19T02:04:09.363 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0028966859999999999 2024-04-19T02:04:09.363 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.363 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.363 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:09.364 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.078218+0000", 2024-04-19T02:04:09.364 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010105799999999999 2024-04-19T02:04:09.364 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.364 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.364 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:09.364 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.078775+0000", 2024-04-19T02:04:09.364 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00055719600000000004 2024-04-19T02:04:09.364 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.364 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.364 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:09.364 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.079292+0000", 2024-04-19T02:04:09.364 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00051654700000000001 2024-04-19T02:04:09.364 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.364 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.364 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:09.365 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.089922+0000", 2024-04-19T02:04:09.365 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010630173999999999 2024-04-19T02:04:09.365 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.365 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.365 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:09.365 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.090036+0000", 2024-04-19T02:04:09.365 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00011389500000000001 2024-04-19T02:04:09.365 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.365 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:09.365 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.365 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.365 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.365 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:756 3.4s0 3:24169e7e:::smithi05930959-31:head [copy-from ver 2] snapc 16=[16,15,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-19T02:04:09.365 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:50.389774+0000", 2024-04-19T02:04:09.365 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 18.938680423000001, 2024-04-19T02:04:09.365 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0130916, 2024-04-19T02:04:09.366 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:09.366 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:09.366 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:09.366 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:09.366 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:09.366 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:09.366 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 756 2024-04-19T02:04:09.366 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.366 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:09.366 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.366 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:09.366 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.389774+0000", 2024-04-19T02:04:09.366 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.366 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.366 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.367 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:09.367 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.389774+0000", 2024-04-19T02:04:09.367 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.367 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.367 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.367 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:09.367 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.389775+0000", 2024-04-19T02:04:09.367 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.604e-06 2024-04-19T02:04:09.367 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.367 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.367 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:09.367 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.389783+0000", 2024-04-19T02:04:09.367 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.1029999999999996e-06 2024-04-19T02:04:09.367 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.367 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.367 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:09.368 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.389796+0000", 2024-04-19T02:04:09.368 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3234999999999999e-05 2024-04-19T02:04:09.368 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.368 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.368 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:09.368 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.389803+0000", 2024-04-19T02:04:09.368 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9619999999999996e-06 2024-04-19T02:04:09.368 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.368 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.368 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.368 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.389876+0000", 2024-04-19T02:04:09.368 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.2898000000000006e-05 2024-04-19T02:04:09.368 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.368 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.368 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:09.368 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.390002+0000", 2024-04-19T02:04:09.369 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000126029 2024-04-19T02:04:09.369 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.369 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.369 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:09.369 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.395353+0000", 2024-04-19T02:04:09.369 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00535156 2024-04-19T02:04:09.369 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.369 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.369 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:09.369 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.396963+0000", 2024-04-19T02:04:09.369 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001609999 2024-04-19T02:04:09.369 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.369 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.369 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:09.369 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.402751+0000", 2024-04-19T02:04:09.370 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0057873000000000004 2024-04-19T02:04:09.370 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.370 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.370 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:09.370 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:50.402865+0000", 2024-04-19T02:04:09.370 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00011491 2024-04-19T02:04:09.370 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.370 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:09.370 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.370 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.370 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.370 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:791 3.4s0 3:208b9382:::smithi05930959-14:head [append 0~802816] snapc 19=[19,17,16,15,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-19T02:04:09.370 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:54.063771+0000", 2024-04-19T02:04:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 15.264683455, 2024-04-19T02:04:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.016213729999999999, 2024-04-19T02:04:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 791 2024-04-19T02:04:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.063771+0000", 2024-04-19T02:04:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.372 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:09.372 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.063771+0000", 2024-04-19T02:04:09.372 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.372 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.372 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.372 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:09.372 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.063774+0000", 2024-04-19T02:04:09.372 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7080000000000002e-06 2024-04-19T02:04:09.372 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.372 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.372 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:09.372 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.064315+0000", 2024-04-19T02:04:09.372 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00054198199999999999 2024-04-19T02:04:09.372 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.372 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.372 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:09.372 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.064337+0000", 2024-04-19T02:04:09.373 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1396e-05 2024-04-19T02:04:09.373 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.373 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.373 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:09.373 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.064352+0000", 2024-04-19T02:04:09.373 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5016e-05 2024-04-19T02:04:09.373 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.373 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.373 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.373 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.064442+0000", 2024-04-19T02:04:09.373 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0204999999999994e-05 2024-04-19T02:04:09.373 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.373 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.373 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "op must wait for map", 2024-04-19T02:04:09.373 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.064450+0000", 2024-04-19T02:04:09.373 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2339999999999994e-06 2024-04-19T02:04:09.373 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.374 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.374 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.374 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.066583+0000", 2024-04-19T02:04:09.374 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0021325799999999998 2024-04-19T02:04:09.374 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.374 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.374 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:09.374 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.066721+0000", 2024-04-19T02:04:09.374 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000138212 2024-04-19T02:04:09.374 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.374 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.374 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:09.374 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.068646+0000", 2024-04-19T02:04:09.374 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0019252060000000001 2024-04-19T02:04:09.375 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.375 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.375 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:09.375 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.079592+0000", 2024-04-19T02:04:09.375 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010945236000000001 2024-04-19T02:04:09.375 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.375 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.375 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:09.375 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.079897+0000", 2024-04-19T02:04:09.375 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00030495499999999998 2024-04-19T02:04:09.375 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.375 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.375 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:09.375 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.079985+0000", 2024-04-19T02:04:09.376 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7999999999999998e-05 2024-04-19T02:04:09.376 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.376 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:09.376 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.376 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.376 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.376 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:792 3.4s0 3:208b9382:::smithi05930959-14:head [append 0~425984] snapc 19=[19,17,16,15,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-19T02:04:09.376 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:54.067320+0000", 2024-04-19T02:04:09.376 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 15.261134577, 2024-04-19T02:04:09.376 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.014750576, 2024-04-19T02:04:09.376 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:09.376 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:09.376 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:09.376 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:09.377 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:09.377 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:09.377 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 792 2024-04-19T02:04:09.377 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.377 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:09.377 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.377 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:09.377 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.067320+0000", 2024-04-19T02:04:09.377 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.377 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.377 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.377 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:09.377 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.067320+0000", 2024-04-19T02:04:09.378 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.378 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.378 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.378 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:09.378 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.067321+0000", 2024-04-19T02:04:09.378 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4810000000000001e-06 2024-04-19T02:04:09.378 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.378 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.378 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:09.378 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.067578+0000", 2024-04-19T02:04:09.378 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00025704100000000001 2024-04-19T02:04:09.378 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.379 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.379 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:09.379 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.067593+0000", 2024-04-19T02:04:09.379 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4942e-05 2024-04-19T02:04:09.379 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.379 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.379 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:09.379 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.067602+0000", 2024-04-19T02:04:09.379 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.2499999999999995e-06 2024-04-19T02:04:09.379 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.379 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.379 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.379 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.069759+0000", 2024-04-19T02:04:09.380 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0021566039999999999 2024-04-19T02:04:09.380 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.380 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.380 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:09.380 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.069916+0000", 2024-04-19T02:04:09.380 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00015739199999999999 2024-04-19T02:04:09.380 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.380 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.380 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:09.380 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.071084+0000", 2024-04-19T02:04:09.380 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0011678529999999999 2024-04-19T02:04:09.380 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.380 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.381 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:09.381 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.079594+0000", 2024-04-19T02:04:09.381 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0085100899999999997 2024-04-19T02:04:09.381 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.381 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.381 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:09.381 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.081979+0000", 2024-04-19T02:04:09.381 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002384227 2024-04-19T02:04:09.381 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.381 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.381 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:09.381 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.082070+0000", 2024-04-19T02:04:09.381 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.1695999999999998e-05 2024-04-19T02:04:09.381 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.381 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:09.382 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.382 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.382 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.382 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:793 3.4s0 3:208b9382:::smithi05930959-14:head [append 0~720896] snapc 19=[19,17,16,15,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-19T02:04:09.382 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:54.072502+0000", 2024-04-19T02:04:09.382 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 15.255951828000001, 2024-04-19T02:04:09.382 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012523014000000001, 2024-04-19T02:04:09.382 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:09.382 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:09.382 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:09.382 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:09.382 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:09.382 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:09.382 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 793 2024-04-19T02:04:09.383 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.383 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:09.383 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.383 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:09.383 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.072502+0000", 2024-04-19T02:04:09.383 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.383 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.383 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.383 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:09.383 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.072502+0000", 2024-04-19T02:04:09.383 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.383 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.383 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.383 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:09.383 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.072505+0000", 2024-04-19T02:04:09.384 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.463e-06 2024-04-19T02:04:09.384 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.384 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.384 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:09.384 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.072987+0000", 2024-04-19T02:04:09.384 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00048223299999999999 2024-04-19T02:04:09.384 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.384 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.384 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:09.384 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.073008+0000", 2024-04-19T02:04:09.384 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.0394000000000002e-05 2024-04-19T02:04:09.384 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.384 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.384 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:09.385 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.073016+0000", 2024-04-19T02:04:09.385 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7350000000000002e-06 2024-04-19T02:04:09.385 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.385 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.385 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.385 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.073224+0000", 2024-04-19T02:04:09.385 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00020745 2024-04-19T02:04:09.385 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.385 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.385 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:09.385 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.073342+0000", 2024-04-19T02:04:09.385 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000117977 2024-04-19T02:04:09.385 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.385 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.385 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:09.386 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.074731+0000", 2024-04-19T02:04:09.386 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0013894980000000001 2024-04-19T02:04:09.386 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.386 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.386 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:09.386 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.079698+0000", 2024-04-19T02:04:09.386 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0049664849999999996 2024-04-19T02:04:09.386 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.386 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.386 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:09.386 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.084909+0000", 2024-04-19T02:04:09.386 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0052113410000000004 2024-04-19T02:04:09.386 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.386 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.387 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:09.387 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.085025+0000", 2024-04-19T02:04:09.387 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000116438 2024-04-19T02:04:09.387 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.387 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:09.387 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.387 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.387 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.387 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:796 3.4s0 3:208b9382:::smithi05930959-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e70)", 2024-04-19T02:04:09.387 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:54.075784+0000", 2024-04-19T02:04:09.387 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 15.252670417999999, 2024-04-19T02:04:09.387 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.014250453999999999, 2024-04-19T02:04:09.387 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:09.387 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:09.388 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:04:09.388 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:09.388 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:09.388 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:09.388 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 796 2024-04-19T02:04:09.388 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.388 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:09.388 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.388 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:09.388 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.075784+0000", 2024-04-19T02:04:09.388 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.388 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.388 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.388 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:09.388 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.075784+0000", 2024-04-19T02:04:09.389 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.389 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.389 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.389 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:09.389 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.075784+0000", 2024-04-19T02:04:09.389 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.84e-07 2024-04-19T02:04:09.389 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.389 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.389 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:09.389 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.075787+0000", 2024-04-19T02:04:09.389 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0180000000000002e-06 2024-04-19T02:04:09.389 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.390 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.390 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:09.390 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.075793+0000", 2024-04-19T02:04:09.390 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.1500000000000004e-06 2024-04-19T02:04:09.390 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.390 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.390 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:09.390 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.075798+0000", 2024-04-19T02:04:09.390 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9049999999999996e-06 2024-04-19T02:04:09.390 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.390 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.390 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.390 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.079433+0000", 2024-04-19T02:04:09.390 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.003634244 2024-04-19T02:04:09.391 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.391 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.391 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-04-19T02:04:09.391 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.079580+0000", 2024-04-19T02:04:09.391 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014772099999999999 2024-04-19T02:04:09.391 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.391 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.391 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.391 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.085902+0000", 2024-04-19T02:04:09.391 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0063216269999999998 2024-04-19T02:04:09.391 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.391 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.391 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:09.391 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.085973+0000", 2024-04-19T02:04:09.392 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.1021999999999999e-05 2024-04-19T02:04:09.392 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.392 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.392 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:09.392 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.090034+0000", 2024-04-19T02:04:09.392 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.004061383 2024-04-19T02:04:09.392 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.392 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:09.392 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.392 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.392 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.392 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:816 3.4s0 3:2859204c:::smithi05930959-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e70)", 2024-04-19T02:04:09.392 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:54.546267+0000", 2024-04-19T02:04:09.392 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 14.782187645, 2024-04-19T02:04:09.392 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012804909, 2024-04-19T02:04:09.393 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:09.393 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:09.393 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:04:09.393 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:09.393 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:09.393 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:09.393 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 816 2024-04-19T02:04:09.393 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.393 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:09.393 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.393 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:09.393 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.546267+0000", 2024-04-19T02:04:09.393 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.394 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.394 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.394 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:09.394 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.546267+0000", 2024-04-19T02:04:09.394 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.394 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.394 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.394 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:09.394 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.546267+0000", 2024-04-19T02:04:09.394 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0399999999999996e-07 2024-04-19T02:04:09.394 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.394 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.394 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:09.394 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.546271+0000", 2024-04-19T02:04:09.394 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.941e-06 2024-04-19T02:04:09.395 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.395 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.395 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:09.395 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.546290+0000", 2024-04-19T02:04:09.395 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8847000000000001e-05 2024-04-19T02:04:09.395 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.395 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.395 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:09.395 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.546299+0000", 2024-04-19T02:04:09.395 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0010000000000007e-06 2024-04-19T02:04:09.395 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.395 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.396 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.396 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.553065+0000", 2024-04-19T02:04:09.396 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0067663050000000002 2024-04-19T02:04:09.396 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.396 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.396 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for rw locks", 2024-04-19T02:04:09.396 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.553155+0000", 2024-04-19T02:04:09.396 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.9289000000000005e-05 2024-04-19T02:04:09.396 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.396 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.396 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.396 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.556246+0000", 2024-04-19T02:04:09.396 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0030913609999999999 2024-04-19T02:04:09.396 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.396 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.397 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:09.397 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.556468+0000", 2024-04-19T02:04:09.397 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000222184 2024-04-19T02:04:09.397 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.397 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.397 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:09.397 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:54.559072+0000", 2024-04-19T02:04:09.397 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002603377 2024-04-19T02:04:09.397 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.397 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:09.397 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.397 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.397 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.397 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:860 3.4s0 3:24169e7e:::smithi05930959-31:head [append 0~0] snapc 1c=[1c,1b,19,17,16,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-04-19T02:04:09.398 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:59.117430+0000", 2024-04-19T02:04:09.398 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 10.211024176, 2024-04-19T02:04:09.398 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012807170999999999, 2024-04-19T02:04:09.398 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:09.398 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:09.398 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:09.398 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:09.398 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:09.398 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:09.398 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 860 2024-04-19T02:04:09.398 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.398 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:09.398 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.399 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:09.399 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.117430+0000", 2024-04-19T02:04:09.399 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.399 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.399 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.399 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:09.399 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.117430+0000", 2024-04-19T02:04:09.399 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.399 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.399 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.399 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:09.399 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.117432+0000", 2024-04-19T02:04:09.399 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4049999999999998e-06 2024-04-19T02:04:09.399 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.399 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.400 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:09.400 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.117444+0000", 2024-04-19T02:04:09.400 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1602e-05 2024-04-19T02:04:09.400 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.400 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.400 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:09.400 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.117473+0000", 2024-04-19T02:04:09.400 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8909999999999999e-05 2024-04-19T02:04:09.400 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.400 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.400 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:09.400 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.117487+0000", 2024-04-19T02:04:09.400 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3647e-05 2024-04-19T02:04:09.400 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.400 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.401 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.401 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.117602+0000", 2024-04-19T02:04:09.401 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00011539700000000001 2024-04-19T02:04:09.401 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.401 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.401 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "op must wait for map", 2024-04-19T02:04:09.401 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.117611+0000", 2024-04-19T02:04:09.401 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.9879999999999993e-06 2024-04-19T02:04:09.401 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.401 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.401 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.401 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.120349+0000", 2024-04-19T02:04:09.401 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0027379940000000001 2024-04-19T02:04:09.401 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.402 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.402 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:09.402 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.120448+0000", 2024-04-19T02:04:09.402 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9195000000000001e-05 2024-04-19T02:04:09.402 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.402 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.402 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:09.402 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.121342+0000", 2024-04-19T02:04:09.402 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00089404299999999996 2024-04-19T02:04:09.402 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.402 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.402 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:09.402 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.125294+0000", 2024-04-19T02:04:09.402 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0039519079999999996 2024-04-19T02:04:09.402 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.403 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.403 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:09.403 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.130170+0000", 2024-04-19T02:04:09.403 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.004875898 2024-04-19T02:04:09.403 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.403 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.403 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:09.403 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.130237+0000", 2024-04-19T02:04:09.403 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.7183999999999999e-05 2024-04-19T02:04:09.403 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.403 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:09.403 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.403 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.403 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.403 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:861 3.4s0 3:24169e7e:::smithi05930959-31:head [rollback 17] snapc 1c=[1c,1b,19,17,16,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-04-19T02:04:09.403 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:59.117524+0000", 2024-04-19T02:04:09.403 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 10.210930079000001, 2024-04-19T02:04:09.404 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012921152, 2024-04-19T02:04:09.404 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:09.404 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:09.404 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:09.404 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:09.404 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:09.404 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:09.404 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 861 2024-04-19T02:04:09.404 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.404 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:09.404 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.404 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:09.404 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.117524+0000", 2024-04-19T02:04:09.404 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.405 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.405 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.405 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:09.405 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.117524+0000", 2024-04-19T02:04:09.405 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.405 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.405 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.405 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:09.405 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.117525+0000", 2024-04-19T02:04:09.405 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.8100000000000001e-07 2024-04-19T02:04:09.405 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.405 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.405 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:09.405 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.117531+0000", 2024-04-19T02:04:09.405 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5600000000000001e-06 2024-04-19T02:04:09.406 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.406 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.406 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:09.406 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.117555+0000", 2024-04-19T02:04:09.406 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4105999999999999e-05 2024-04-19T02:04:09.406 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.406 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.406 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:09.406 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.117566+0000", 2024-04-19T02:04:09.406 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1117999999999999e-05 2024-04-19T02:04:09.406 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.406 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.406 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.406 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.117681+0000", 2024-04-19T02:04:09.407 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000115463 2024-04-19T02:04:09.407 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.407 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.407 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:04:09.407 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.117689+0000", 2024-04-19T02:04:09.407 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.8520000000000004e-06 2024-04-19T02:04:09.407 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.407 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.407 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.407 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.122192+0000", 2024-04-19T02:04:09.407 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0045033080000000001 2024-04-19T02:04:09.407 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.407 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.407 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:09.407 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.122335+0000", 2024-04-19T02:04:09.408 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014219600000000001 2024-04-19T02:04:09.408 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.408 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.408 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:09.408 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.123193+0000", 2024-04-19T02:04:09.408 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00085811299999999997 2024-04-19T02:04:09.408 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.408 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.408 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:09.408 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.125296+0000", 2024-04-19T02:04:09.408 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002102746 2024-04-19T02:04:09.408 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.408 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.408 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:09.408 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.130364+0000", 2024-04-19T02:04:09.409 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0050681379999999998 2024-04-19T02:04:09.409 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.409 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.409 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:09.409 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.130445+0000", 2024-04-19T02:04:09.409 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.1670999999999998e-05 2024-04-19T02:04:09.409 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.409 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:09.409 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.409 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.409 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.409 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:862 3.4s0 3:24169e7e:::smithi05930959-31:head [append 0~0] snapc 1c=[1c,1b,19,17,16,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-04-19T02:04:09.409 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:59.117592+0000", 2024-04-19T02:04:09.409 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 10.21086182, 2024-04-19T02:04:09.410 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.015889105000000001, 2024-04-19T02:04:09.410 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:09.410 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:09.410 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:09.410 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:09.410 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:09.410 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:09.410 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 862 2024-04-19T02:04:09.410 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.410 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:09.410 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.410 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:09.410 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.117592+0000", 2024-04-19T02:04:09.410 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.411 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.411 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.411 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:09.411 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.117592+0000", 2024-04-19T02:04:09.411 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.411 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.411 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.411 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:09.411 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.117593+0000", 2024-04-19T02:04:09.411 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0699999999999997e-07 2024-04-19T02:04:09.411 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.411 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.411 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:09.412 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.117597+0000", 2024-04-19T02:04:09.412 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6739999999999999e-06 2024-04-19T02:04:09.412 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.412 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.412 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:09.412 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.117605+0000", 2024-04-19T02:04:09.412 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0730000000000005e-06 2024-04-19T02:04:09.412 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.412 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.412 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:09.412 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.117611+0000", 2024-04-19T02:04:09.412 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.1580000000000002e-06 2024-04-19T02:04:09.413 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.413 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.413 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.413 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.117751+0000", 2024-04-19T02:04:09.413 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014034400000000001 2024-04-19T02:04:09.413 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.413 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.413 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:04:09.413 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.117761+0000", 2024-04-19T02:04:09.413 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9229999999999997e-06 2024-04-19T02:04:09.413 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.413 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.413 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.413 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.123655+0000", 2024-04-19T02:04:09.414 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0058939090000000001 2024-04-19T02:04:09.414 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.414 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.414 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:09.414 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.123796+0000", 2024-04-19T02:04:09.414 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014073499999999999 2024-04-19T02:04:09.414 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.414 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.414 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:09.414 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.124414+0000", 2024-04-19T02:04:09.414 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00061841499999999998 2024-04-19T02:04:09.414 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.414 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.414 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:09.415 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.126163+0000", 2024-04-19T02:04:09.415 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001748954 2024-04-19T02:04:09.415 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.415 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.415 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:09.415 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.133388+0000", 2024-04-19T02:04:09.415 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0072248190000000004 2024-04-19T02:04:09.415 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.415 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.415 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:09.415 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:59.133482+0000", 2024-04-19T02:04:09.415 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3493999999999996e-05 2024-04-19T02:04:09.415 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.415 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:09.415 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.416 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.416 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.416 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:869 3.4s0 3:2859204c:::smithi05930959-39:head [create,setxattr gtfaOn]^VZb\\e^pjgmXO_SNbaZXkiniU\\P`oQPhU (278) in=40b,setxattr dpZPTbk (1857) in=7b,setxattr R^V_S_WOsbWmoeWSdPX\\YOo`p`PYTj`h_ReacUgt (665) in=40b,setxattr QZmj (1857) in=4b,setxattr tm_[dr`m_TOd`sSpak`PmkiibcrmTV_XSn^WjshQ (1929) in=40b,setxattr ]bj[s]anfnljkN\\a^\\kZbtkqmPdS]a[r]RYpZOof (385) in=40b,setxattr QmbcqkkYZeUsOV_XRmqlpbXsojT\\rjjhpaXbktY] (17824) in=40b,setxattr Nj^kdhftmNdm]r]aPOh^lkcehSWXqcYR[[QYfOce (15156) in=40b,setxattr ^r]XUeieXbq`VhtQSso`sWWVnSn^Rlgk`iQPa\\ra (10596) in=40b,setxattr ^Z_pYXoiQZenjnnhgOjfrXNRa^URaTTSUm`lfRcg (630) in=40b,setxattr `\\Wrragae (1857) in=9b,setxattr QgltNjUn^_`XqfqoNt\\TXaqYsojWjTOZsm`lso^g (139) in=40b,setxattr SjOaT`ghYmoqcYgh]NalVZ\\VQ`mbPli`ZeafRcn] (1434) in=40b,setxattr lONdXUcf`cijsY^pPb`QQkPfn]NgSTkpY`RYgogY (3576) in=40b,setxattr rZY`[g]ml^N[YqhZZgUgVTjUYomnaR^dqenTO^a] (349) in=40b,setxattr ]_fjc]_tsQa_O]Ot^bT\\oR]UnXrQ^tRlObcUsj^\\ (1331) in=40b,setxattr OUteY^]pdhrp`f[VVXhV_sSiQ_scQefbfb_neloX (346) in=40b,setxattr htrV^k`d\\kZcaVbdXlfPcQfVZO\\rknjrqnkTYpft (1400) in=40b,setxattr jo[[[sY_`WQZbc\\Q\\oReWPfthXVP (1857) in=28b,setxattr TnirX]edWgqQj\\frs[embprebRY[dp]q`lPaRQgW (803) in=40b] snapc 1d=[1d,1c,1b,19,17,16,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-04-19T02:04:09.416 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:04:00.124478+0000", 2024-04-19T02:04:09.416 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 9.2039766440000008, 2024-04-19T02:04:09.416 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.014590928, 2024-04-19T02:04:09.416 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:09.416 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:09.416 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:09.416 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:09.416 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:09.416 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:09.416 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 869 2024-04-19T02:04:09.416 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.417 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:09.417 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.417 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:09.417 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.124478+0000", 2024-04-19T02:04:09.417 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.417 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.417 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.417 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:09.417 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.124478+0000", 2024-04-19T02:04:09.417 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.417 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.417 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.417 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:09.417 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.124481+0000", 2024-04-19T02:04:09.417 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.371e-06 2024-04-19T02:04:09.418 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.418 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.418 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:09.418 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.124526+0000", 2024-04-19T02:04:09.418 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5395000000000002e-05 2024-04-19T02:04:09.418 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.418 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.418 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:09.418 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.124548+0000", 2024-04-19T02:04:09.418 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.1719999999999999e-05 2024-04-19T02:04:09.418 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.418 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.418 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:09.418 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.124563+0000", 2024-04-19T02:04:09.419 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4565e-05 2024-04-19T02:04:09.419 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.419 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.419 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.419 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.124659+0000", 2024-04-19T02:04:09.419 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.6343999999999998e-05 2024-04-19T02:04:09.419 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.419 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.419 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "op must wait for map", 2024-04-19T02:04:09.419 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.124668+0000", 2024-04-19T02:04:09.419 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7369999999999995e-06 2024-04-19T02:04:09.419 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.419 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.419 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.419 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.128929+0000", 2024-04-19T02:04:09.420 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0042610690000000001 2024-04-19T02:04:09.420 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.420 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.420 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:09.420 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.129074+0000", 2024-04-19T02:04:09.420 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014543499999999999 2024-04-19T02:04:09.420 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.420 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.420 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:09.420 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.130505+0000", 2024-04-19T02:04:09.420 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0014308490000000001 2024-04-19T02:04:09.420 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.420 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.420 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:09.421 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.132303+0000", 2024-04-19T02:04:09.421 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001797589 2024-04-19T02:04:09.421 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.421 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.421 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:09.421 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.138911+0000", 2024-04-19T02:04:09.421 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0066082780000000004 2024-04-19T02:04:09.421 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.421 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.421 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:09.421 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.139069+0000", 2024-04-19T02:04:09.421 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00015757600000000001 2024-04-19T02:04:09.421 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.421 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:09.421 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.422 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.422 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.422 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:875 3.4s0 3:24202781:::smithi05930959-26:head [append 0~0] snapc 1d=[1d,1c,1b,19,17,16,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-04-19T02:04:09.422 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:04:00.149480+0000", 2024-04-19T02:04:09.422 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 9.1789742909999994, 2024-04-19T02:04:09.422 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.015807758000000002, 2024-04-19T02:04:09.422 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:09.422 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:09.422 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:09.422 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:09.422 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:09.422 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:09.422 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 875 2024-04-19T02:04:09.423 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.423 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:09.423 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.423 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:09.423 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.149480+0000", 2024-04-19T02:04:09.423 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.423 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.423 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.423 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:09.423 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.149480+0000", 2024-04-19T02:04:09.423 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.423 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.423 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.424 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:09.424 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.149482+0000", 2024-04-19T02:04:09.424 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.024e-06 2024-04-19T02:04:09.424 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.424 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.424 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:09.424 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.149493+0000", 2024-04-19T02:04:09.424 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1314e-05 2024-04-19T02:04:09.424 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.424 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.424 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:09.424 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.149520+0000", 2024-04-19T02:04:09.424 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6696e-05 2024-04-19T02:04:09.424 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.425 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.425 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:09.425 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.149530+0000", 2024-04-19T02:04:09.425 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.7969999999999995e-06 2024-04-19T02:04:09.425 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.425 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.425 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.425 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.149601+0000", 2024-04-19T02:04:09.425 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.1648000000000002e-05 2024-04-19T02:04:09.425 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.425 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.425 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:09.425 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.149771+0000", 2024-04-19T02:04:09.425 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00016909600000000001 2024-04-19T02:04:09.425 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.426 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.426 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:09.426 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.150669+0000", 2024-04-19T02:04:09.426 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00089854500000000005 2024-04-19T02:04:09.426 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.426 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.426 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:09.426 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.154559+0000", 2024-04-19T02:04:09.426 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0038896209999999998 2024-04-19T02:04:09.426 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.426 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.426 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:09.426 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.165217+0000", 2024-04-19T02:04:09.427 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0106579 2024-04-19T02:04:09.427 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.427 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.427 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:09.427 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.165288+0000", 2024-04-19T02:04:09.427 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.1117000000000004e-05 2024-04-19T02:04:09.427 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.427 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:09.427 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.427 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.427 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.427 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:876 3.4s0 3:24202781:::smithi05930959-26:head [rollback 19] snapc 1d=[1d,1c,1b,19,17,16,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-04-19T02:04:09.427 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:04:00.149556+0000", 2024-04-19T02:04:09.427 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 9.1788984249999999, 2024-04-19T02:04:09.427 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.027244712000000001, 2024-04-19T02:04:09.428 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:09.428 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:09.428 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:09.428 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:09.428 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:09.428 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:09.428 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 876 2024-04-19T02:04:09.428 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.428 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:09.428 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.428 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:09.428 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.149556+0000", 2024-04-19T02:04:09.428 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.428 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.429 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.429 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:09.429 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.149556+0000", 2024-04-19T02:04:09.429 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.429 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.429 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.429 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:09.429 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.149556+0000", 2024-04-19T02:04:09.429 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5299999999999999e-07 2024-04-19T02:04:09.429 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.429 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.429 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:09.429 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.149559+0000", 2024-04-19T02:04:09.429 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8200000000000001e-06 2024-04-19T02:04:09.429 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.429 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.430 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:09.430 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.149567+0000", 2024-04-19T02:04:09.430 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.5379999999999997e-06 2024-04-19T02:04:09.430 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.430 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.430 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:09.430 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.149575+0000", 2024-04-19T02:04:09.430 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.9039999999999992e-06 2024-04-19T02:04:09.430 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.430 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.430 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.430 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.151277+0000", 2024-04-19T02:04:09.430 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0017027959999999999 2024-04-19T02:04:09.431 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.431 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.431 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:09.431 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.151362+0000", 2024-04-19T02:04:09.431 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5014999999999998e-05 2024-04-19T02:04:09.431 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.431 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.431 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:09.431 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.152028+0000", 2024-04-19T02:04:09.431 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00066526200000000001 2024-04-19T02:04:09.431 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.431 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.431 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:09.431 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.154564+0000", 2024-04-19T02:04:09.431 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0025368119999999998 2024-04-19T02:04:09.431 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.432 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.432 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:09.432 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.176673+0000", 2024-04-19T02:04:09.432 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.022108243 2024-04-19T02:04:09.432 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.432 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.432 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:09.432 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.176801+0000", 2024-04-19T02:04:09.432 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00012786899999999999 2024-04-19T02:04:09.432 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.432 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:09.432 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.432 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.432 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.432 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:877 3.4s0 3:24202781:::smithi05930959-26:head [append 0~0] snapc 1d=[1d,1c,1b,19,17,16,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-04-19T02:04:09.433 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:04:00.149591+0000", 2024-04-19T02:04:09.433 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 9.1788636379999993, 2024-04-19T02:04:09.433 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.027440069000000001, 2024-04-19T02:04:09.433 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:09.433 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:09.433 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:09.433 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:09.433 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:09.433 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:09.433 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 877 2024-04-19T02:04:09.433 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.433 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:09.433 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.433 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:09.433 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.149591+0000", 2024-04-19T02:04:09.434 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.434 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.434 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.434 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:09.434 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.149591+0000", 2024-04-19T02:04:09.434 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.434 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.434 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.434 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:09.434 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.149591+0000", 2024-04-19T02:04:09.434 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.9400000000000005e-07 2024-04-19T02:04:09.434 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.434 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.434 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:09.434 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.149595+0000", 2024-04-19T02:04:09.435 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1620000000000001e-06 2024-04-19T02:04:09.435 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.435 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.435 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:09.435 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.149605+0000", 2024-04-19T02:04:09.435 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.5070000000000006e-06 2024-04-19T02:04:09.435 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.435 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.435 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:09.435 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.149613+0000", 2024-04-19T02:04:09.435 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.7659999999999998e-06 2024-04-19T02:04:09.435 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.435 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.435 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.435 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.152371+0000", 2024-04-19T02:04:09.436 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0027586450000000001 2024-04-19T02:04:09.436 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.436 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.436 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:09.436 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.152494+0000", 2024-04-19T02:04:09.436 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00012304800000000001 2024-04-19T02:04:09.436 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.436 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.436 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:09.436 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.153093+0000", 2024-04-19T02:04:09.436 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00059848599999999996 2024-04-19T02:04:09.436 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.436 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.436 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:09.436 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.155947+0000", 2024-04-19T02:04:09.436 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002854325 2024-04-19T02:04:09.437 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.437 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.437 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:09.437 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.176938+0000", 2024-04-19T02:04:09.437 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.020990981999999998 2024-04-19T02:04:09.437 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.437 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.437 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:09.437 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.177031+0000", 2024-04-19T02:04:09.437 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.2553999999999995e-05 2024-04-19T02:04:09.437 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.437 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:09.437 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.437 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.437 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:09.438 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:04:09.439 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-19T02:04:09.440 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:04:09.441+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:04:09.452 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 reset_purged_snaps_last 2024-04-19T02:04:09.540 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:04:09.541+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:04:09.625 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:04:09.625 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:04:09.625 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-04-19T02:04:09.625 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:04:09.641 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-19T02:04:09.641 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:04:09.641+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:04:09.671 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 reset_purged_snaps_last 2024-04-19T02:04:09.742 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-04-19T02:04:09.741+0000 7feed2045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25039) UID: 0 2024-04-19T02:04:09.826 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:04:09.827 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:04:09.827 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-04-19T02:04:09.827 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-04-19T02:04:09.827 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:04:09.841 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-19T02:04:09.842 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:04:09.845+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:04:09.869 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 reset_purged_snaps_last 2024-04-19T02:04:09.955 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:04:09.955 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-04-19T02:04:09.955 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-04-19T02:04:09.955 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-04-19T02:04:09.955 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.955 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 40/26 ECSubWrite(tid=30, reqid=client.4509.0:359, at_version=40'16, trim_to=0'0, roll_forward_to=39'14))", 2024-04-19T02:04:09.955 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.790646+0000", 2024-04-19T02:04:09.955 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 45.165726820000003, 2024-04-19T02:04:09.955 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.004294422, 2024-04-19T02:04:09.955 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:09.956 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:09.956 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:09.956 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:09.956 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.956 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:09.956 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.790646+0000", 2024-04-19T02:04:09.956 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.956 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.956 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.956 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:09.956 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.790646+0000", 2024-04-19T02:04:09.956 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.956 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.957 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.957 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:09.957 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.790647+0000", 2024-04-19T02:04:09.957 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7400000000000003e-07 2024-04-19T02:04:09.957 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.957 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.957 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:09.957 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.791098+0000", 2024-04-19T02:04:09.957 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00045124599999999998 2024-04-19T02:04:09.957 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.957 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.957 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:09.957 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.791116+0000", 2024-04-19T02:04:09.957 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7351999999999999e-05 2024-04-19T02:04:09.958 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.958 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.958 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:09.958 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.791126+0000", 2024-04-19T02:04:09.958 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0606e-05 2024-04-19T02:04:09.958 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.958 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.958 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.958 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.791206+0000", 2024-04-19T02:04:09.958 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0018999999999994e-05 2024-04-19T02:04:09.958 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.958 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.959 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:09.959 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.791217+0000", 2024-04-19T02:04:09.959 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0361e-05 2024-04-19T02:04:09.959 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.959 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.959 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:09.959 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.794897+0000", 2024-04-19T02:04:09.959 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0036802810000000001 2024-04-19T02:04:09.959 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.959 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.959 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:09.959 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.794941+0000", 2024-04-19T02:04:09.959 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.3983000000000002e-05 2024-04-19T02:04:09.959 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.960 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:09.960 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.960 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.960 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.960 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s5 42/26 ECSubWrite(tid=43, reqid=osd.1.0:43, at_version=42'30, trim_to=0'0, roll_forward_to=40'28))", 2024-04-19T02:04:09.960 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:26.737676+0000", 2024-04-19T02:04:09.960 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 43.218697155999998, 2024-04-19T02:04:09.960 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0079882579999999998, 2024-04-19T02:04:09.960 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:09.960 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:09.960 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:09.960 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:09.960 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.960 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:09.961 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.737676+0000", 2024-04-19T02:04:09.961 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.961 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.961 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.961 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:09.961 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.737676+0000", 2024-04-19T02:04:09.961 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.961 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.961 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.961 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:09.961 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.737676+0000", 2024-04-19T02:04:09.961 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3999999999999997e-07 2024-04-19T02:04:09.961 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.961 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.962 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:09.962 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.737712+0000", 2024-04-19T02:04:09.962 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5405999999999999e-05 2024-04-19T02:04:09.962 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.962 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.962 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:09.962 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.737731+0000", 2024-04-19T02:04:09.962 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9437e-05 2024-04-19T02:04:09.962 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.962 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.962 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:09.962 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.737746+0000", 2024-04-19T02:04:09.963 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5013e-05 2024-04-19T02:04:09.963 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.963 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.963 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.963 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.737887+0000", 2024-04-19T02:04:09.963 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014084300000000001 2024-04-19T02:04:09.963 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.963 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.963 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:09.963 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.737902+0000", 2024-04-19T02:04:09.963 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5138e-05 2024-04-19T02:04:09.963 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.963 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.964 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:09.964 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.745612+0000", 2024-04-19T02:04:09.964 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0077100049999999998 2024-04-19T02:04:09.964 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.964 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.964 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:09.964 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.745664+0000", 2024-04-19T02:04:09.964 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.2076000000000001e-05 2024-04-19T02:04:09.964 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.964 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:09.964 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.964 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.964 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.965 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s5 42/26 ECSubWrite(tid=44, reqid=osd.1.0:44, at_version=42'32, trim_to=0'0, roll_forward_to=40'28))", 2024-04-19T02:04:09.965 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:26.739008+0000", 2024-04-19T02:04:09.965 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 43.217365549, 2024-04-19T02:04:09.965 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0066868379999999996, 2024-04-19T02:04:09.965 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:09.965 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:09.965 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:09.965 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:09.965 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.965 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:09.965 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.739008+0000", 2024-04-19T02:04:09.965 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.965 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.965 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.965 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:09.966 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.739008+0000", 2024-04-19T02:04:09.966 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.966 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.966 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.966 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:09.966 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.739009+0000", 2024-04-19T02:04:09.966 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.4300000000000001e-07 2024-04-19T02:04:09.966 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.966 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.966 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:09.966 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.739041+0000", 2024-04-19T02:04:09.966 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2642999999999999e-05 2024-04-19T02:04:09.966 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.967 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.967 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:09.967 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.739057+0000", 2024-04-19T02:04:09.967 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6124e-05 2024-04-19T02:04:09.967 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.967 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.967 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:09.967 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.739067+0000", 2024-04-19T02:04:09.967 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3389999999999999e-06 2024-04-19T02:04:09.967 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.967 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.967 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.967 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.739141+0000", 2024-04-19T02:04:09.967 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.3794999999999994e-05 2024-04-19T02:04:09.967 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.968 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.968 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:09.968 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.739152+0000", 2024-04-19T02:04:09.968 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0954e-05 2024-04-19T02:04:09.968 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.968 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.968 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:09.968 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.745671+0000", 2024-04-19T02:04:09.968 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0065197450000000004 2024-04-19T02:04:09.968 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.968 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.968 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:09.968 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.745695+0000", 2024-04-19T02:04:09.968 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3295e-05 2024-04-19T02:04:09.969 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.969 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:09.969 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.969 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.969 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.969 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.5s2 42/26 ECSubWrite(tid=51, reqid=osd.10.0:51, at_version=42'27, trim_to=0'0, roll_forward_to=39'25))", 2024-04-19T02:04:09.969 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:26.748457+0000", 2024-04-19T02:04:09.969 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 43.207916181000002, 2024-04-19T02:04:09.969 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0043907169999999997, 2024-04-19T02:04:09.969 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:09.969 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:09.969 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:09.969 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:09.969 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.970 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:09.970 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.748457+0000", 2024-04-19T02:04:09.970 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.970 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.970 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.970 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:09.970 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.748457+0000", 2024-04-19T02:04:09.970 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.970 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.970 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.970 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:09.970 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.748458+0000", 2024-04-19T02:04:09.970 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0699999999999998e-07 2024-04-19T02:04:09.970 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.971 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.971 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:09.971 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.748491+0000", 2024-04-19T02:04:09.971 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.392e-05 2024-04-19T02:04:09.971 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.971 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.971 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:09.971 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.748509+0000", 2024-04-19T02:04:09.971 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7450000000000001e-05 2024-04-19T02:04:09.971 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.971 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.971 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:09.971 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.748521+0000", 2024-04-19T02:04:09.971 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2591e-05 2024-04-19T02:04:09.972 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.972 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.972 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.972 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.748598+0000", 2024-04-19T02:04:09.972 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.6886999999999998e-05 2024-04-19T02:04:09.972 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.972 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.972 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:09.972 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.748608+0000", 2024-04-19T02:04:09.972 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.6709999999999994e-06 2024-04-19T02:04:09.972 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.972 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.972 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:09.972 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.752792+0000", 2024-04-19T02:04:09.972 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0041843339999999996 2024-04-19T02:04:09.973 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.973 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.973 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:09.973 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.752848+0000", 2024-04-19T02:04:09.973 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5457000000000001e-05 2024-04-19T02:04:09.973 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.973 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:09.973 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.973 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.973 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.973 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.as7 42/26 ECSubWrite(tid=82, reqid=osd.11.0:82, at_version=42'34, trim_to=0'0, roll_forward_to=39'32))", 2024-04-19T02:04:09.973 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:26.757483+0000", 2024-04-19T02:04:09.973 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 43.198890546999998, 2024-04-19T02:04:09.974 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0054335750000000004, 2024-04-19T02:04:09.974 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:09.974 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:09.974 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:09.974 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:09.974 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.974 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:09.974 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757483+0000", 2024-04-19T02:04:09.974 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.974 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.974 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.974 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:09.974 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757483+0000", 2024-04-19T02:04:09.975 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.975 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.975 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.975 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:09.975 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757483+0000", 2024-04-19T02:04:09.975 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1e-07 2024-04-19T02:04:09.975 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.975 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.975 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:09.975 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757508+0000", 2024-04-19T02:04:09.975 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4681999999999998e-05 2024-04-19T02:04:09.975 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.975 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.975 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:09.975 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757525+0000", 2024-04-19T02:04:09.976 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6869000000000001e-05 2024-04-19T02:04:09.976 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.976 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.976 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:09.976 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757535+0000", 2024-04-19T02:04:09.976 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0074999999999999e-05 2024-04-19T02:04:09.976 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.976 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.976 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.976 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757714+0000", 2024-04-19T02:04:09.976 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000179388 2024-04-19T02:04:09.976 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.976 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.976 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:09.977 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757729+0000", 2024-04-19T02:04:09.977 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4703e-05 2024-04-19T02:04:09.977 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.977 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.977 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:09.977 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.762889+0000", 2024-04-19T02:04:09.977 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0051599430000000002 2024-04-19T02:04:09.977 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.977 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.977 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:09.977 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.762916+0000", 2024-04-19T02:04:09.977 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7605000000000001e-05 2024-04-19T02:04:09.977 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.977 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:09.977 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.978 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.978 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.978 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.fs2 42/26 ECSubWrite(tid=83, reqid=osd.11.0:83, at_version=42'26, trim_to=0'0, roll_forward_to=39'24))", 2024-04-19T02:04:09.978 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:26.757561+0000", 2024-04-19T02:04:09.978 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 43.198812668999999, 2024-04-19T02:04:09.978 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0053118139999999998, 2024-04-19T02:04:09.978 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:09.978 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:09.978 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:09.978 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:09.978 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.978 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:09.978 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757561+0000", 2024-04-19T02:04:09.979 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.979 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.979 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.979 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:09.979 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757561+0000", 2024-04-19T02:04:09.979 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.979 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.979 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.979 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:09.979 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757561+0000", 2024-04-19T02:04:09.979 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.58e-07 2024-04-19T02:04:09.979 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.979 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.979 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:09.980 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757602+0000", 2024-04-19T02:04:09.980 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.0790000000000001e-05 2024-04-19T02:04:09.980 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.980 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.980 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:09.980 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757612+0000", 2024-04-19T02:04:09.980 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0078000000000001e-05 2024-04-19T02:04:09.980 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.980 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.980 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:09.980 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757648+0000", 2024-04-19T02:04:09.980 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.6046e-05 2024-04-19T02:04:09.980 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.980 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.981 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.981 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757718+0000", 2024-04-19T02:04:09.981 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9772999999999998e-05 2024-04-19T02:04:09.981 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.981 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.981 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:09.981 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.757727+0000", 2024-04-19T02:04:09.981 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.7280000000000007e-06 2024-04-19T02:04:09.981 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.981 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.981 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:09.981 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.762829+0000", 2024-04-19T02:04:09.981 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0051016539999999997 2024-04-19T02:04:09.982 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.982 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.982 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:09.982 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.762872+0000", 2024-04-19T02:04:09.982 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.3387000000000003e-05 2024-04-19T02:04:09.982 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.982 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:09.982 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.982 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.982 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.982 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 42/26 ECSubWrite(tid=32, reqid=osd.9.0:32, at_version=42'18, trim_to=0'0, roll_forward_to=40'16))", 2024-04-19T02:04:09.982 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:26.765866+0000", 2024-04-19T02:04:09.982 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 43.190507052999997, 2024-04-19T02:04:09.982 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0081652840000000001, 2024-04-19T02:04:09.983 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:09.983 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:09.983 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:09.983 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:09.983 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.983 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:09.983 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.765866+0000", 2024-04-19T02:04:09.983 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.983 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.983 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.983 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:09.983 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.765866+0000", 2024-04-19T02:04:09.983 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.983 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.984 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.984 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:09.984 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.765867+0000", 2024-04-19T02:04:09.984 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7599999999999998e-07 2024-04-19T02:04:09.984 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.984 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.984 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:09.984 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.765890+0000", 2024-04-19T02:04:09.984 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3459e-05 2024-04-19T02:04:09.984 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.984 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.984 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:09.984 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.765901+0000", 2024-04-19T02:04:09.985 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1426e-05 2024-04-19T02:04:09.985 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.985 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.985 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:09.985 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.765911+0000", 2024-04-19T02:04:09.985 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.2969999999999992e-06 2024-04-19T02:04:09.985 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.985 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.985 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.985 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.765996+0000", 2024-04-19T02:04:09.985 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.5568999999999993e-05 2024-04-19T02:04:09.985 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.985 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.985 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:09.986 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.766011+0000", 2024-04-19T02:04:09.986 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4615e-05 2024-04-19T02:04:09.986 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.986 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.986 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:09.986 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.773989+0000", 2024-04-19T02:04:09.986 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0079778669999999996 2024-04-19T02:04:09.986 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.986 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.986 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:09.986 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:26.774032+0000", 2024-04-19T02:04:09.986 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2775000000000003e-05 2024-04-19T02:04:09.986 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.987 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:09.987 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.987 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.987 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.987 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 46/26 ECSubWrite(tid=44, reqid=osd.9.0:44, at_version=46'26, trim_to=0'0, roll_forward_to=45'24))", 2024-04-19T02:04:09.987 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.065734+0000", 2024-04-19T02:04:09.987 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 39.890639284999999, 2024-04-19T02:04:09.987 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0090304430000000008, 2024-04-19T02:04:09.987 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:09.987 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:09.987 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:09.987 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:09.987 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.987 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:09.988 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.065734+0000", 2024-04-19T02:04:09.988 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.988 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.988 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.988 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:09.988 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.065734+0000", 2024-04-19T02:04:09.988 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.988 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.988 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.988 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:09.988 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.065734+0000", 2024-04-19T02:04:09.988 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2399999999999999e-07 2024-04-19T02:04:09.988 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.988 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.989 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:09.989 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.065763+0000", 2024-04-19T02:04:09.989 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8393000000000001e-05 2024-04-19T02:04:09.989 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.989 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.989 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:09.989 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.065786+0000", 2024-04-19T02:04:09.989 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2909e-05 2024-04-19T02:04:09.989 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.989 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.989 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:09.989 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.065799+0000", 2024-04-19T02:04:09.989 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2975e-05 2024-04-19T02:04:09.989 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.990 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.990 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.990 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.069211+0000", 2024-04-19T02:04:09.990 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0034118899999999999 2024-04-19T02:04:09.990 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.990 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.990 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:09.990 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.069234+0000", 2024-04-19T02:04:09.990 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.372e-05 2024-04-19T02:04:09.990 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.990 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.990 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:09.990 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.074695+0000", 2024-04-19T02:04:09.990 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0054611779999999997 2024-04-19T02:04:09.991 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.991 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.991 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:09.991 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.074764+0000", 2024-04-19T02:04:09.991 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8954000000000004e-05 2024-04-19T02:04:09.991 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.991 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:09.991 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.991 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.991 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.991 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.as7 46/26 ECSubWrite(tid=103, reqid=osd.11.0:103, at_version=46'49, trim_to=0'0, roll_forward_to=45'45))", 2024-04-19T02:04:09.991 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.070195+0000", 2024-04-19T02:04:09.991 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 39.886178387999998, 2024-04-19T02:04:09.991 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0048631990000000003, 2024-04-19T02:04:09.991 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:09.992 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:09.992 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:09.992 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:09.992 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.992 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:09.992 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.070195+0000", 2024-04-19T02:04:09.992 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.992 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.992 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.992 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:09.992 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.070195+0000", 2024-04-19T02:04:09.992 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.992 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.992 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.992 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:09.993 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.070196+0000", 2024-04-19T02:04:09.993 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8899999999999999e-07 2024-04-19T02:04:09.993 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.993 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.993 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:09.993 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.070219+0000", 2024-04-19T02:04:09.993 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3485999999999998e-05 2024-04-19T02:04:09.993 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.993 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.993 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:09.993 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.070230+0000", 2024-04-19T02:04:09.993 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1039000000000001e-05 2024-04-19T02:04:09.993 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.993 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.993 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:09.994 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.070241+0000", 2024-04-19T02:04:09.994 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0866999999999999e-05 2024-04-19T02:04:09.994 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.994 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.994 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.994 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.070720+0000", 2024-04-19T02:04:09.994 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00047945099999999999 2024-04-19T02:04:09.994 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.994 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.994 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:09.994 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.070734+0000", 2024-04-19T02:04:09.994 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3811e-05 2024-04-19T02:04:09.994 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.994 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.995 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:09.995 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.075014+0000", 2024-04-19T02:04:09.995 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.004280196 2024-04-19T02:04:09.995 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.995 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.995 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:09.995 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.075058+0000", 2024-04-19T02:04:09.995 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.3659999999999999e-05 2024-04-19T02:04:09.995 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.995 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:09.995 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:09.995 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.995 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.995 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s5 46/26 ECSubWrite(tid=59, reqid=osd.1.0:58, at_version=0'0, trim_to=0'0, roll_forward_to=46'44))", 2024-04-19T02:04:09.995 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.077390+0000", 2024-04-19T02:04:09.996 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 39.878982882999999, 2024-04-19T02:04:09.996 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0043230480000000003, 2024-04-19T02:04:09.996 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:09.996 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:09.996 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:09.996 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:09.996 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.996 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:09.996 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077390+0000", 2024-04-19T02:04:09.996 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.996 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.996 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.996 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:09.996 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077390+0000", 2024-04-19T02:04:09.996 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:09.997 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.997 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.997 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:09.997 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077392+0000", 2024-04-19T02:04:09.997 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2270000000000001e-06 2024-04-19T02:04:09.997 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.997 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.997 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:09.997 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077409+0000", 2024-04-19T02:04:09.997 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7748999999999999e-05 2024-04-19T02:04:09.997 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.997 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.997 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:09.997 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077436+0000", 2024-04-19T02:04:09.998 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6451000000000001e-05 2024-04-19T02:04:09.998 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.998 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.998 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:09.998 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077502+0000", 2024-04-19T02:04:09.998 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.6493000000000002e-05 2024-04-19T02:04:09.998 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.998 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.998 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:09.998 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077638+0000", 2024-04-19T02:04:09.999 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013563599999999999 2024-04-19T02:04:09.999 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.999 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.999 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:09.999 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.077650+0000", 2024-04-19T02:04:09.999 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1569999999999999e-05 2024-04-19T02:04:09.999 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.999 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:09.999 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:09.999 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.081644+0000", 2024-04-19T02:04:09.999 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0039942950000000001 2024-04-19T02:04:09.999 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:09.999 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.000 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.000 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.081713+0000", 2024-04-19T02:04:10.000 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9627000000000003e-05 2024-04-19T02:04:10.000 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.000 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.000 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.000 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.000 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.000 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.es5 46/26 ECSubWrite(tid=39, reqid=client.4509.0:420, at_version=46'31, trim_to=0'0, roll_forward_to=46'29))", 2024-04-19T02:04:10.000 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.240512+0000", 2024-04-19T02:04:10.000 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 39.715861324999999, 2024-04-19T02:04:10.000 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0043966530000000004, 2024-04-19T02:04:10.000 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:10.001 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:10.001 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:10.001 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:10.001 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.001 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:10.001 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.240512+0000", 2024-04-19T02:04:10.001 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.001 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.001 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.001 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:10.001 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.240512+0000", 2024-04-19T02:04:10.001 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.001 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.001 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.002 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:10.002 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.240514+0000", 2024-04-19T02:04:10.002 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.717e-06 2024-04-19T02:04:10.002 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.002 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.002 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:10.002 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.242558+0000", 2024-04-19T02:04:10.002 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0020443940000000002 2024-04-19T02:04:10.002 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.002 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.002 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:10.002 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.242576+0000", 2024-04-19T02:04:10.003 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8127000000000001e-05 2024-04-19T02:04:10.003 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.003 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.003 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:10.003 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.242585+0000", 2024-04-19T02:04:10.003 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.1300000000000007e-06 2024-04-19T02:04:10.003 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.003 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.003 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.003 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.242678+0000", 2024-04-19T02:04:10.003 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.2808999999999999e-05 2024-04-19T02:04:10.003 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.003 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.003 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:10.004 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.242685+0000", 2024-04-19T02:04:10.004 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.7700000000000004e-06 2024-04-19T02:04:10.004 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.004 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.004 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:10.004 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.244876+0000", 2024-04-19T02:04:10.004 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0021906270000000001 2024-04-19T02:04:10.004 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.004 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.004 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.004 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.244909+0000", 2024-04-19T02:04:10.004 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3079e-05 2024-04-19T02:04:10.004 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.005 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.005 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.005 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.005 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.005 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s5 47/26 ECSubWrite(tid=62, reqid=client.4509.0:482, at_version=47'47, trim_to=0'0, roll_forward_to=47'45))", 2024-04-19T02:04:10.005 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:31.216102+0000", 2024-04-19T02:04:10.005 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 38.740271704000001, 2024-04-19T02:04:10.005 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0045766349999999999, 2024-04-19T02:04:10.005 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:10.005 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:10.005 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:10.005 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:10.005 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.005 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:10.006 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.216102+0000", 2024-04-19T02:04:10.006 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.006 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.006 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.006 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:10.006 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.216102+0000", 2024-04-19T02:04:10.006 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.006 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.006 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.006 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:10.006 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.216103+0000", 2024-04-19T02:04:10.006 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.683e-06 2024-04-19T02:04:10.006 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.007 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.007 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:10.007 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.216227+0000", 2024-04-19T02:04:10.007 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00012341500000000001 2024-04-19T02:04:10.007 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.007 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.007 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:10.007 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.216252+0000", 2024-04-19T02:04:10.007 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5525999999999999e-05 2024-04-19T02:04:10.007 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.007 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.007 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:10.007 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.216263+0000", 2024-04-19T02:04:10.007 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1273e-05 2024-04-19T02:04:10.008 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.008 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.008 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.008 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.216367+0000", 2024-04-19T02:04:10.008 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000103427 2024-04-19T02:04:10.008 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.008 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.008 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:10.008 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.216378+0000", 2024-04-19T02:04:10.008 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0684e-05 2024-04-19T02:04:10.008 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.008 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.008 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:10.008 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.220623+0000", 2024-04-19T02:04:10.008 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0042451870000000001 2024-04-19T02:04:10.009 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.009 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.009 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.009 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:31.220678+0000", 2024-04-19T02:04:10.009 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5439999999999998e-05 2024-04-19T02:04:10.009 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.009 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.009 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.009 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.009 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.009 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.cs5 51/26 ECSubWrite(tid=52, reqid=osd.9.0:51, at_version=0'0, trim_to=0'0, roll_forward_to=51'30))", 2024-04-19T02:04:10.009 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:35.116856+0000", 2024-04-19T02:04:10.009 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 34.839517196999999, 2024-04-19T02:04:10.009 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0055490599999999998, 2024-04-19T02:04:10.010 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:10.010 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:10.010 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:10.010 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:10.010 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.010 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:10.010 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:35.116856+0000", 2024-04-19T02:04:10.010 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.010 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.010 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.010 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:10.010 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:35.116856+0000", 2024-04-19T02:04:10.010 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.010 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.011 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.011 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:10.011 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:35.116857+0000", 2024-04-19T02:04:10.011 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0800000000000004e-07 2024-04-19T02:04:10.011 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.011 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.011 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:10.011 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:35.116871+0000", 2024-04-19T02:04:10.011 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.456e-05 2024-04-19T02:04:10.011 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.011 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.011 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:10.011 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:35.116884+0000", 2024-04-19T02:04:10.011 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2374999999999999e-05 2024-04-19T02:04:10.011 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.012 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.012 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:10.012 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:35.116894+0000", 2024-04-19T02:04:10.012 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.026e-05 2024-04-19T02:04:10.012 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.012 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.012 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.012 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:35.116986+0000", 2024-04-19T02:04:10.012 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.2021999999999994e-05 2024-04-19T02:04:10.012 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.012 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.012 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:10.012 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:35.117001+0000", 2024-04-19T02:04:10.012 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5235999999999999e-05 2024-04-19T02:04:10.013 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.013 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.013 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:10.013 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:35.122349+0000", 2024-04-19T02:04:10.013 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0053481400000000004 2024-04-19T02:04:10.013 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.013 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.013 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.013 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:35.122405+0000", 2024-04-19T02:04:10.013 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5658999999999997e-05 2024-04-19T02:04:10.013 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.013 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.013 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.013 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.014 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.014 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s5 57/26 ECSubWrite(tid=71, reqid=osd.1.0:71, at_version=57'56, trim_to=0'0, roll_forward_to=55'54))", 2024-04-19T02:04:10.014 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:41.141947+0000", 2024-04-19T02:04:10.014 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 28.814425809999999, 2024-04-19T02:04:10.014 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0043941919999999999, 2024-04-19T02:04:10.014 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:10.014 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:10.014 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:10.014 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:10.014 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.014 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:10.014 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.141947+0000", 2024-04-19T02:04:10.014 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.014 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.015 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.015 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:10.015 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.141947+0000", 2024-04-19T02:04:10.015 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.015 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.015 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.015 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:10.015 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.141948+0000", 2024-04-19T02:04:10.015 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.3700000000000001e-07 2024-04-19T02:04:10.015 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.015 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.015 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:10.015 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.141986+0000", 2024-04-19T02:04:10.015 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.7758e-05 2024-04-19T02:04:10.016 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.016 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.016 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:10.016 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.142020+0000", 2024-04-19T02:04:10.016 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4146999999999997e-05 2024-04-19T02:04:10.016 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.016 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.016 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:10.016 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.142036+0000", 2024-04-19T02:04:10.016 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5726000000000002e-05 2024-04-19T02:04:10.016 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.016 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.016 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.016 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.142223+0000", 2024-04-19T02:04:10.017 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00018726600000000001 2024-04-19T02:04:10.017 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.017 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.017 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:10.017 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.142241+0000", 2024-04-19T02:04:10.017 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.8413000000000001e-05 2024-04-19T02:04:10.017 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.017 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.017 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:10.017 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.146291+0000", 2024-04-19T02:04:10.017 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0040499999999999998 2024-04-19T02:04:10.017 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.017 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.017 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.018 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.146342+0000", 2024-04-19T02:04:10.018 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.0445000000000002e-05 2024-04-19T02:04:10.018 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.018 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.018 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.018 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.018 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.018 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.as7 57/26 ECSubWrite(tid=136, reqid=osd.11.0:136, at_version=57'62, trim_to=0'0, roll_forward_to=55'60))", 2024-04-19T02:04:10.018 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:41.154457+0000", 2024-04-19T02:04:10.018 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 28.801916213999998, 2024-04-19T02:04:10.018 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0055113590000000004, 2024-04-19T02:04:10.019 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:10.019 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:10.019 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:10.019 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:10.019 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.019 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:10.019 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.154457+0000", 2024-04-19T02:04:10.019 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.019 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.019 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.019 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:10.020 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.154457+0000", 2024-04-19T02:04:10.020 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.020 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.020 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.020 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:10.020 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.154457+0000", 2024-04-19T02:04:10.020 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2300000000000002e-07 2024-04-19T02:04:10.020 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.020 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.020 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:10.020 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.154489+0000", 2024-04-19T02:04:10.020 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1301999999999999e-05 2024-04-19T02:04:10.020 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.021 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.021 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:10.021 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.154506+0000", 2024-04-19T02:04:10.021 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7554000000000002e-05 2024-04-19T02:04:10.021 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.021 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.021 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:10.021 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.154519+0000", 2024-04-19T02:04:10.021 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3074000000000001e-05 2024-04-19T02:04:10.021 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.021 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.021 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.021 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.154628+0000", 2024-04-19T02:04:10.022 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108974 2024-04-19T02:04:10.022 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.022 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.022 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:10.022 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.154653+0000", 2024-04-19T02:04:10.022 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4207999999999999e-05 2024-04-19T02:04:10.022 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.022 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.022 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:10.022 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.159911+0000", 2024-04-19T02:04:10.022 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0052585059999999996 2024-04-19T02:04:10.022 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.023 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.023 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.023 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.159968+0000", 2024-04-19T02:04:10.023 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7417999999999998e-05 2024-04-19T02:04:10.023 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.023 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.023 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.023 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.023 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.023 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s5 57/26 ECSubWrite(tid=72, reqid=osd.1.0:71, at_version=0'0, trim_to=0'0, roll_forward_to=57'56))", 2024-04-19T02:04:10.023 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:41.156487+0000", 2024-04-19T02:04:10.023 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 28.799886397000002, 2024-04-19T02:04:10.023 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0058785720000000003, 2024-04-19T02:04:10.023 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:10.023 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:10.024 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:10.024 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:10.024 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.024 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:10.024 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.156487+0000", 2024-04-19T02:04:10.024 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.024 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.024 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.024 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:10.024 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.156487+0000", 2024-04-19T02:04:10.024 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.024 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.024 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.024 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:10.025 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.156488+0000", 2024-04-19T02:04:10.025 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2449999999999999e-06 2024-04-19T02:04:10.025 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.025 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.025 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:10.025 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.156511+0000", 2024-04-19T02:04:10.025 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.2429000000000002e-05 2024-04-19T02:04:10.025 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.025 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.025 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:10.025 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.156530+0000", 2024-04-19T02:04:10.025 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9389999999999999e-05 2024-04-19T02:04:10.025 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.025 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.025 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:10.025 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.156543+0000", 2024-04-19T02:04:10.026 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2676e-05 2024-04-19T02:04:10.026 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.026 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.026 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.026 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.156876+0000", 2024-04-19T02:04:10.026 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00033304599999999998 2024-04-19T02:04:10.026 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.026 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.026 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:10.026 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.156891+0000", 2024-04-19T02:04:10.026 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4921e-05 2024-04-19T02:04:10.026 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.026 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.027 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:10.027 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.162307+0000", 2024-04-19T02:04:10.027 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.005416227 2024-04-19T02:04:10.027 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.027 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.027 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.027 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:41.162365+0000", 2024-04-19T02:04:10.027 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.8638000000000003e-05 2024-04-19T02:04:10.027 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.027 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.027 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.027 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.027 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.027 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.5s2 62/26 ECSubWrite(tid=78, reqid=client.4509.0:703, at_version=62'35, trim_to=0'0, roll_forward_to=45'33))", 2024-04-19T02:04:10.028 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:46.470518+0000", 2024-04-19T02:04:10.028 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 23.485854888999999, 2024-04-19T02:04:10.028 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0042076220000000003, 2024-04-19T02:04:10.028 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:10.028 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:10.028 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:10.028 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:10.028 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.028 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:10.028 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.470518+0000", 2024-04-19T02:04:10.028 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.028 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.028 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.028 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:10.029 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.470518+0000", 2024-04-19T02:04:10.029 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.029 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.029 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.029 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:10.029 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.470519+0000", 2024-04-19T02:04:10.029 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2000000000000001e-07 2024-04-19T02:04:10.029 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.029 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.029 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:10.029 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.470690+0000", 2024-04-19T02:04:10.029 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00017127200000000001 2024-04-19T02:04:10.029 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.029 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.030 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:10.030 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.470714+0000", 2024-04-19T02:04:10.030 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.357e-05 2024-04-19T02:04:10.030 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.030 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.030 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:10.030 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.470726+0000", 2024-04-19T02:04:10.030 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2272e-05 2024-04-19T02:04:10.030 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.030 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.030 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.030 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.470859+0000", 2024-04-19T02:04:10.030 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00013266999999999999 2024-04-19T02:04:10.030 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.031 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.031 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:10.031 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.470869+0000", 2024-04-19T02:04:10.031 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0838e-05 2024-04-19T02:04:10.031 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.031 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.031 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:10.031 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.474672+0000", 2024-04-19T02:04:10.031 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.003802911 2024-04-19T02:04:10.031 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.031 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.031 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.031 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:46.474726+0000", 2024-04-19T02:04:10.031 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.3769e-05 2024-04-19T02:04:10.031 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.032 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.032 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.032 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.032 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.032 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.5s2 67/26 ECSubWrite(tid=85, reqid=client.4509.0:765, at_version=67'41, trim_to=0'0, roll_forward_to=64'39))", 2024-04-19T02:04:10.032 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:51.097679+0000", 2024-04-19T02:04:10.032 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 18.858693852999998, 2024-04-19T02:04:10.032 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0042775929999999997, 2024-04-19T02:04:10.032 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:10.032 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:10.032 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:10.032 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:10.032 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.032 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:10.032 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:51.097679+0000", 2024-04-19T02:04:10.033 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.033 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.033 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.033 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:10.033 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:51.097679+0000", 2024-04-19T02:04:10.033 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.033 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.033 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.033 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:10.033 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:51.097680+0000", 2024-04-19T02:04:10.033 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.22e-07 2024-04-19T02:04:10.033 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.033 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.033 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:10.033 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:51.098288+0000", 2024-04-19T02:04:10.034 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00060858400000000001 2024-04-19T02:04:10.034 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.034 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.034 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:10.034 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:51.098304+0000", 2024-04-19T02:04:10.034 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6030999999999999e-05 2024-04-19T02:04:10.034 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.034 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.034 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:10.034 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:51.098315+0000", 2024-04-19T02:04:10.034 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1056000000000001e-05 2024-04-19T02:04:10.034 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.034 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.034 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.035 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:51.098435+0000", 2024-04-19T02:04:10.035 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000119743 2024-04-19T02:04:10.035 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.035 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.035 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:10.035 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:51.098449+0000", 2024-04-19T02:04:10.035 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3750999999999999e-05 2024-04-19T02:04:10.035 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.035 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.035 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:10.035 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:51.101907+0000", 2024-04-19T02:04:10.035 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.003458544 2024-04-19T02:04:10.035 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.035 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.036 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.036 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:51.101957+0000", 2024-04-19T02:04:10.036 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9561999999999997e-05 2024-04-19T02:04:10.036 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.036 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.036 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.036 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.036 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.036 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.as7 76/26 ECSubWrite(tid=194, reqid=client.4509.0:884, at_version=76'86, trim_to=0'0, roll_forward_to=73'84))", 2024-04-19T02:04:10.036 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:04:00.216904+0000", 2024-04-19T02:04:10.036 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 9.7394696120000006, 2024-04-19T02:04:10.036 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0086969809999999995, 2024-04-19T02:04:10.036 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:10.036 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:10.036 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:10.037 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:10.037 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.037 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:10.037 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.216904+0000", 2024-04-19T02:04:10.037 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.037 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.037 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.037 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:10.037 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.216904+0000", 2024-04-19T02:04:10.037 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.037 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.037 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.037 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:10.037 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.216905+0000", 2024-04-19T02:04:10.037 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6929999999999999e-06 2024-04-19T02:04:10.038 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.038 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.038 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:10.038 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.221110+0000", 2024-04-19T02:04:10.038 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.004204246 2024-04-19T02:04:10.038 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.038 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.038 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:10.038 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.221134+0000", 2024-04-19T02:04:10.038 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.425e-05 2024-04-19T02:04:10.038 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.038 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.038 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:10.038 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.221146+0000", 2024-04-19T02:04:10.038 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2463000000000001e-05 2024-04-19T02:04:10.039 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.039 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.039 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.039 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.221251+0000", 2024-04-19T02:04:10.039 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010493300000000001 2024-04-19T02:04:10.039 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.039 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.039 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:10.039 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.221268+0000", 2024-04-19T02:04:10.039 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6690000000000001e-05 2024-04-19T02:04:10.039 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.039 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.039 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:10.039 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.225562+0000", 2024-04-19T02:04:10.040 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0042940410000000002 2024-04-19T02:04:10.040 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.040 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.040 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.040 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.225601+0000", 2024-04-19T02:04:10.040 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.8665000000000003e-05 2024-04-19T02:04:10.040 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.040 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.040 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.040 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.040 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.040 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWrite(3.4s5 80/26 ECSubWrite(tid=136, reqid=osd.1.0:136, at_version=80'119, trim_to=0'0, roll_forward_to=79'117))", 2024-04-19T02:04:10.040 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:04:04.152792+0000", 2024-04-19T02:04:10.040 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 5.8035810770000005, 2024-04-19T02:04:10.040 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0043869820000000002, 2024-04-19T02:04:10.041 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:10.041 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:10.041 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:10.041 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:10.041 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.041 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:10.041 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:04.152792+0000", 2024-04-19T02:04:10.041 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.041 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.041 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.041 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:10.041 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:04.152792+0000", 2024-04-19T02:04:10.041 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.041 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.042 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.042 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:10.042 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:04.152793+0000", 2024-04-19T02:04:10.042 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5600000000000001e-07 2024-04-19T02:04:10.042 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.042 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.042 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:10.042 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:04.152838+0000", 2024-04-19T02:04:10.042 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5303000000000003e-05 2024-04-19T02:04:10.042 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.042 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.042 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:10.042 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:04.152865+0000", 2024-04-19T02:04:10.042 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.6812999999999999e-05 2024-04-19T02:04:10.043 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.043 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.043 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:10.043 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:04.152880+0000", 2024-04-19T02:04:10.043 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5058999999999999e-05 2024-04-19T02:04:10.043 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.043 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.043 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.043 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:04.153049+0000", 2024-04-19T02:04:10.043 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000169715 2024-04-19T02:04:10.043 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.043 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.043 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:10.043 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:04.153065+0000", 2024-04-19T02:04:10.043 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5594e-05 2024-04-19T02:04:10.044 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.044 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.044 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:10.044 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:04.157128+0000", 2024-04-19T02:04:10.044 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0040633830000000003 2024-04-19T02:04:10.044 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.044 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.044 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.044 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:04.157179+0000", 2024-04-19T02:04:10.044 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.0759e-05 2024-04-19T02:04:10.044 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.044 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.044 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.044 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.044 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.045 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:04:10.045 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:04:09.953+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:04:10.046 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-19T02:04:10.069 INFO:tasks.rados.rados.0.smithi059.stdout:374: done (0 left) 2024-04-19T02:04:10.069 INFO:tasks.rados.rados.0.smithi059.stdout:375: read oid 10 snap -1 2024-04-19T02:04:10.069 INFO:tasks.rados.rados.0.smithi059.stdout:375: expect (ObjNum 88 snap 9 seq_num 88) 2024-04-19T02:04:10.069 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 39 current snap is 33 2024-04-19T02:04:10.069 INFO:tasks.rados.rados.0.smithi059.stdout:376: seq_num 162 ranges {3588096=327680} 2024-04-19T02:04:10.069 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:04:10.069+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:04:10.085 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 reset_purged_snaps_last 2024-04-19T02:04:10.093 INFO:tasks.rados.rados.0.smithi059.stdout:376: writing smithi05930959-39 from 3588096 to 3915776 tid 1 2024-04-19T02:04:10.093 INFO:tasks.rados.rados.0.smithi059.stdout:377: setattr oid 29 current snap is 33 2024-04-19T02:04:10.094 INFO:tasks.rados.rados.0.smithi059.stdout:375: expect (ObjNum 88 snap 9 seq_num 88) 2024-04-19T02:04:10.148 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:04:10.149+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:04:10.163 INFO:tasks.rados.rados.0.smithi059.stdout:376: finishing write tid 1 to smithi05930959-39 2024-04-19T02:04:10.163 INFO:tasks.rados.rados.0.smithi059.stdout:376: finishing write tid 2 to smithi05930959-39 2024-04-19T02:04:10.163 INFO:tasks.rados.rados.0.smithi059.stdout:376: finishing write tid 3 to smithi05930959-39 2024-04-19T02:04:10.163 INFO:tasks.rados.rados.0.smithi059.stdout:376: oid 39 updating version 0 to 139 2024-04-19T02:04:10.164 INFO:tasks.rados.rados.0.smithi059.stdout:376: oid 39 updating version 139 to 140 2024-04-19T02:04:10.164 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 39 v 140 (ObjNum 162 snap 33 seq_num 162) dirty exists 2024-04-19T02:04:10.164 INFO:tasks.rados.rados.0.smithi059.stdout:376: left oid 39 (ObjNum 162 snap 33 seq_num 162) 2024-04-19T02:04:10.164 INFO:tasks.rados.rados.0.smithi059.stdout:375: done (2 left) 2024-04-19T02:04:10.164 INFO:tasks.rados.rados.0.smithi059.stdout:376: done (1 left) 2024-04-19T02:04:10.164 INFO:tasks.rados.rados.0.smithi059.stdout:378: delete oid 39 current snap is 33 2024-04-19T02:04:10.169 INFO:tasks.rados.rados.0.smithi059.stdout:378: done (1 left) 2024-04-19T02:04:10.169 INFO:tasks.rados.rados.0.smithi059.stdout:379: rmattr oid 41 current snap is 33 2024-04-19T02:04:10.169 INFO:tasks.rados.rados.0.smithi059.stdout:379: done (1 left) 2024-04-19T02:04:10.169 INFO:tasks.rados.rados.0.smithi059.stdout:380: setattr oid 9 current snap is 33 2024-04-19T02:04:10.170 INFO:tasks.rados.rados.0.smithi059.stdout:381: delete oid 50 current snap is 33 2024-04-19T02:04:10.177 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 29 v 67 (ObjNum 138 snap 22 seq_num 138) dirty exists 2024-04-19T02:04:10.186 INFO:tasks.rados.rados.0.smithi059.stdout:377: done (2 left) 2024-04-19T02:04:10.186 INFO:tasks.rados.rados.0.smithi059.stdout:381: done (1 left) 2024-04-19T02:04:10.186 INFO:tasks.rados.rados.0.smithi059.stdout:382: rmattr oid 26 current snap is 33 2024-04-19T02:04:10.186 INFO:tasks.rados.rados.0.smithi059.stdout:382: done (1 left) 2024-04-19T02:04:10.186 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 17 current snap is 33 2024-04-19T02:04:10.187 INFO:tasks.rados.rados.0.smithi059.stdout:383: seq_num 165 ranges {0=376832} 2024-04-19T02:04:10.190 INFO:tasks.rados.rados.0.smithi059.stdout:383: writing smithi05930959-17 from 0 to 376832 tid 1 2024-04-19T02:04:10.190 INFO:tasks.rados.rados.0.smithi059.stdout:384: copy_from oid 3 from oid 46 current snap is 33 2024-04-19T02:04:10.190 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 9 v 144 (ObjNum 33 snap 0 seq_num 1244372896) dirty exists 2024-04-19T02:04:10.190 INFO:tasks.rados.rados.0.smithi059.stdout:380: done (2 left) 2024-04-19T02:04:10.190 INFO:tasks.rados.rados.0.smithi059.stdout:385: read oid 40 snap -1 2024-04-19T02:04:10.190 INFO:tasks.rados.rados.0.smithi059.stdout:385: expect (ObjNum 142 snap 24 seq_num 142) 2024-04-19T02:04:10.190 INFO:tasks.rados.rados.0.smithi059.stdout:386: read oid 43 snap 32 2024-04-19T02:04:10.190 INFO:tasks.rados.rados.0.smithi059.stdout:386: expect (ObjNum 1243376800 snap 21854 seq_num 1243869200) 2024-04-19T02:04:10.190 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 35 current snap is 33 2024-04-19T02:04:10.190 INFO:tasks.rados.rados.0.smithi059.stdout:387: seq_num 166 ranges {573440=393216} 2024-04-19T02:04:10.196 INFO:tasks.rados.rados.0.smithi059.stdout:387: writing smithi05930959-35 from 573440 to 966656 tid 1 2024-04-19T02:04:10.196 INFO:tasks.rados.rados.0.smithi059.stdout:388: snap_remove snap 25 2024-04-19T02:04:10.203 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:04:10.203 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:04:10.203 INFO:teuthology.orchestra.run.smithi059.stdout: "num_ops": 0 2024-04-19T02:04:10.203 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:04:10.221 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-19T02:04:10.265 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 reset_purged_snaps_last 2024-04-19T02:04:10.266 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:04:10.265+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:04:10.337 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:04:10.337 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [], 2024-04-19T02:04:10.337 INFO:teuthology.orchestra.run.smithi059.stdout: "complaint_time": 30, 2024-04-19T02:04:10.337 INFO:teuthology.orchestra.run.smithi059.stdout: "num_blocked_ops": 0 2024-04-19T02:04:10.337 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:04:10.350 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:04:10.349+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:04:10.351 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-19T02:04:10.451 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:04:10.449+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:04:10.524 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:04:10.524 INFO:teuthology.orchestra.run.smithi059.stdout: "size": 20, 2024-04-19T02:04:10.524 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 600, 2024-04-19T02:04:10.525 INFO:teuthology.orchestra.run.smithi059.stdout: "ops": [ 2024-04-19T02:04:10.525 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.525 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:238 3.2s0 3:407ecba7:::smithi05930959-22:2 [read 0~360448,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-19T02:04:10.525 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.646510+0000", 2024-04-19T02:04:10.525 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 53.879158095000001, 2024-04-19T02:04:10.525 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.028744658999999999, 2024-04-19T02:04:10.525 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:10.525 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:10.525 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:04:10.525 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:10.525 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:10.525 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:10.525 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 238 2024-04-19T02:04:10.525 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.526 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:10.526 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.526 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:10.526 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646510+0000", 2024-04-19T02:04:10.526 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.526 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.526 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.526 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:10.526 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646510+0000", 2024-04-19T02:04:10.526 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.526 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.526 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.526 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:10.527 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646511+0000", 2024-04-19T02:04:10.527 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.8599999999999997e-07 2024-04-19T02:04:10.527 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.527 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.527 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:10.527 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646517+0000", 2024-04-19T02:04:10.527 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.1700000000000002e-06 2024-04-19T02:04:10.527 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.527 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.527 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:10.527 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646528+0000", 2024-04-19T02:04:10.527 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1202999999999999e-05 2024-04-19T02:04:10.527 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.527 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.528 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:10.528 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646537+0000", 2024-04-19T02:04:10.528 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7460000000000006e-06 2024-04-19T02:04:10.528 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.528 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.528 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.528 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.647105+0000", 2024-04-19T02:04:10.528 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00056801599999999996 2024-04-19T02:04:10.528 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.528 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.528 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:04:10.528 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.647127+0000", 2024-04-19T02:04:10.528 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.175e-05 2024-04-19T02:04:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.657504+0000", 2024-04-19T02:04:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.010377961 2024-04-19T02:04:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.657826+0000", 2024-04-19T02:04:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00032128300000000001 2024-04-19T02:04:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.675254+0000", 2024-04-19T02:04:10.529 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.017428644 2024-04-19T02:04:10.530 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.530 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.530 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.530 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.530 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.530 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:239 3.2s0 3:407ecba7:::smithi05930959-22:2 [read 0~360448,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-19T02:04:10.530 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:16.646549+0000", 2024-04-19T02:04:10.530 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 53.879118624, 2024-04-19T02:04:10.530 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.029381570999999999, 2024-04-19T02:04:10.530 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:10.530 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:10.530 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:04:10.530 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:10.531 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:10.531 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:10.531 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 239 2024-04-19T02:04:10.531 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.531 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:10.531 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.531 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:10.531 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646549+0000", 2024-04-19T02:04:10.531 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.531 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.531 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.531 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:10.531 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646549+0000", 2024-04-19T02:04:10.531 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.531 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.532 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.532 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:10.532 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646550+0000", 2024-04-19T02:04:10.532 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.8400000000000004e-07 2024-04-19T02:04:10.532 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.532 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.532 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:10.532 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646555+0000", 2024-04-19T02:04:10.532 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9989999999999999e-06 2024-04-19T02:04:10.532 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.532 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.532 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:10.532 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646567+0000", 2024-04-19T02:04:10.532 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.202e-05 2024-04-19T02:04:10.533 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.533 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.533 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:10.533 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.646575+0000", 2024-04-19T02:04:10.533 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.7640000000000005e-06 2024-04-19T02:04:10.533 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.533 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.533 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.533 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.647229+0000", 2024-04-19T02:04:10.533 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00065415299999999998 2024-04-19T02:04:10.533 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.533 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.533 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:04:10.533 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.647255+0000", 2024-04-19T02:04:10.534 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.5795999999999998e-05 2024-04-19T02:04:10.534 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.534 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.534 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.534 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.659835+0000", 2024-04-19T02:04:10.534 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012580713 2024-04-19T02:04:10.534 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.534 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.534 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:10.534 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.660090+0000", 2024-04-19T02:04:10.534 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00025430899999999998 2024-04-19T02:04:10.534 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.534 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.534 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.535 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:16.675931+0000", 2024-04-19T02:04:10.535 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.015841133 2024-04-19T02:04:10.535 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.535 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.535 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.535 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.535 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.535 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:349 3.2s0 3:43b7722e:::smithi05930959-10:head [copy-from ver 11] snapc 8=[8,6,5] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:04:10.535 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.746333+0000", 2024-04-19T02:04:10.535 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 45.779334466999998, 2024-04-19T02:04:10.536 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.034219922999999999, 2024-04-19T02:04:10.536 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:10.536 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:10.536 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:10.536 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:10.536 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:10.536 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:10.536 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 349 2024-04-19T02:04:10.536 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.536 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:10.536 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.536 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:10.536 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746333+0000", 2024-04-19T02:04:10.536 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.536 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.537 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.537 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:10.537 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746333+0000", 2024-04-19T02:04:10.537 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.537 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.537 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.537 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:10.537 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746337+0000", 2024-04-19T02:04:10.537 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4769999999999999e-06 2024-04-19T02:04:10.537 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.537 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.537 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:10.537 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746352+0000", 2024-04-19T02:04:10.537 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5277e-05 2024-04-19T02:04:10.538 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.538 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.538 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:10.538 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746387+0000", 2024-04-19T02:04:10.538 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4436e-05 2024-04-19T02:04:10.538 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.538 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.538 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:10.538 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746403+0000", 2024-04-19T02:04:10.538 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.6475999999999999e-05 2024-04-19T02:04:10.538 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.538 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.538 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.539 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746490+0000", 2024-04-19T02:04:10.539 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.7213000000000007e-05 2024-04-19T02:04:10.539 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.539 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.539 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "op must wait for map", 2024-04-19T02:04:10.539 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746497+0000", 2024-04-19T02:04:10.539 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.0520000000000004e-06 2024-04-19T02:04:10.539 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.539 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.539 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.539 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749865+0000", 2024-04-19T02:04:10.539 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0033676069999999999 2024-04-19T02:04:10.539 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.540 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.540 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:10.540 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.749938+0000", 2024-04-19T02:04:10.540 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.3530999999999998e-05 2024-04-19T02:04:10.540 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.540 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.540 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:10.540 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.768855+0000", 2024-04-19T02:04:10.540 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.018916493 2024-04-19T02:04:10.540 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.540 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.540 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:10.540 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.771377+0000", 2024-04-19T02:04:10.540 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0025221520000000002 2024-04-19T02:04:10.541 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.541 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.541 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:10.541 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.780460+0000", 2024-04-19T02:04:10.541 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0090830389999999993 2024-04-19T02:04:10.541 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.541 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.541 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.541 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.780553+0000", 2024-04-19T02:04:10.541 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3170000000000004e-05 2024-04-19T02:04:10.541 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.541 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.541 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.541 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.542 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.542 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:350 3.2s0 3:43b7722e:::smithi05930959-10:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-04-19T02:04:10.542 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:24.746450+0000", 2024-04-19T02:04:10.542 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 45.779217807999999, 2024-04-19T02:04:10.542 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.024575331999999998, 2024-04-19T02:04:10.542 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:10.542 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:10.542 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:04:10.542 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:10.542 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:10.542 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:10.542 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 350 2024-04-19T02:04:10.542 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.542 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:10.543 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.543 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:10.543 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746450+0000", 2024-04-19T02:04:10.543 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.543 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.543 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.543 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:10.543 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746450+0000", 2024-04-19T02:04:10.543 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.543 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.543 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.543 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:10.543 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746451+0000", 2024-04-19T02:04:10.543 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.0400000000000005e-07 2024-04-19T02:04:10.543 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.544 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.544 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:10.544 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746456+0000", 2024-04-19T02:04:10.544 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.5199999999999997e-06 2024-04-19T02:04:10.544 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.544 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.544 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:10.544 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746470+0000", 2024-04-19T02:04:10.544 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3935e-05 2024-04-19T02:04:10.544 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.544 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.544 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:10.544 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746481+0000", 2024-04-19T02:04:10.544 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0919e-05 2024-04-19T02:04:10.544 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.545 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.545 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.545 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746552+0000", 2024-04-19T02:04:10.545 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.0235999999999996e-05 2024-04-19T02:04:10.545 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.545 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.545 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:04:10.545 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.746557+0000", 2024-04-19T02:04:10.545 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.6570000000000002e-06 2024-04-19T02:04:10.545 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.545 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.545 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.545 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750143+0000", 2024-04-19T02:04:10.545 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0035860549999999999 2024-04-19T02:04:10.546 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.546 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.546 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for blocked object", 2024-04-19T02:04:10.546 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.750177+0000", 2024-04-19T02:04:10.546 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3796e-05 2024-04-19T02:04:10.546 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.546 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.546 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.546 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.770842+0000", 2024-04-19T02:04:10.546 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.020664657999999999 2024-04-19T02:04:10.546 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.546 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.546 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:10.546 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.770923+0000", 2024-04-19T02:04:10.546 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0869000000000001e-05 2024-04-19T02:04:10.547 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.547 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.547 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.547 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:24.771025+0000", 2024-04-19T02:04:10.547 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010278299999999999 2024-04-19T02:04:10.547 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.547 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.547 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.547 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.547 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.547 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:422 3.2s0 3:43b7722e:::smithi05930959-10:head [copy-from ver 11] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:10.547 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.193952+0000", 2024-04-19T02:04:10.547 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 40.331715469000002, 2024-04-19T02:04:10.547 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.044114190999999997, 2024-04-19T02:04:10.547 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:10.548 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:10.548 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:10.548 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:10.548 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:10.548 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:10.548 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 422 2024-04-19T02:04:10.548 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.548 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:10.548 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.548 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:10.548 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.193952+0000", 2024-04-19T02:04:10.548 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.548 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.548 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.548 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:10.549 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.193952+0000", 2024-04-19T02:04:10.549 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.549 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.549 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.549 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:10.549 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.193955+0000", 2024-04-19T02:04:10.549 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.3159999999999999e-06 2024-04-19T02:04:10.549 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.549 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.549 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:10.549 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.193968+0000", 2024-04-19T02:04:10.549 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3726999999999999e-05 2024-04-19T02:04:10.549 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.549 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.549 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:10.549 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.193993+0000", 2024-04-19T02:04:10.550 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4499e-05 2024-04-19T02:04:10.550 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.550 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.550 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:10.550 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194005+0000", 2024-04-19T02:04:10.550 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1770000000000001e-05 2024-04-19T02:04:10.550 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.550 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.550 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.550 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194109+0000", 2024-04-19T02:04:10.550 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000103904 2024-04-19T02:04:10.550 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.550 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.550 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:10.550 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194249+0000", 2024-04-19T02:04:10.551 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00014036699999999999 2024-04-19T02:04:10.551 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.551 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.551 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:10.551 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.224308+0000", 2024-04-19T02:04:10.551 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.030059210999999999 2024-04-19T02:04:10.551 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.551 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.551 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:10.551 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.237172+0000", 2024-04-19T02:04:10.551 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012863734999999999 2024-04-19T02:04:10.551 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.551 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.551 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:10.551 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.237230+0000", 2024-04-19T02:04:10.552 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.8238e-05 2024-04-19T02:04:10.552 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.552 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.552 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.552 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.238067+0000", 2024-04-19T02:04:10.552 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000836424 2024-04-19T02:04:10.552 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.552 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.552 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.552 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.552 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.552 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:423 3.2s0 3:43b7722e:::smithi05930959-10:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:10.552 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194040+0000", 2024-04-19T02:04:10.552 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 40.331628139000003, 2024-04-19T02:04:10.553 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.040466800999999997, 2024-04-19T02:04:10.553 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:10.553 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:10.553 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:04:10.553 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:10.553 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:10.553 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:10.553 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 423 2024-04-19T02:04:10.553 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.553 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:10.553 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.553 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:10.553 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194040+0000", 2024-04-19T02:04:10.553 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.554 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.554 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.554 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:10.554 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194040+0000", 2024-04-19T02:04:10.554 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.554 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.554 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.554 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:10.554 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194040+0000", 2024-04-19T02:04:10.554 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.4700000000000001e-07 2024-04-19T02:04:10.554 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.554 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.554 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:10.554 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194044+0000", 2024-04-19T02:04:10.555 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.9049999999999999e-06 2024-04-19T02:04:10.555 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.555 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.555 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:10.555 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194056+0000", 2024-04-19T02:04:10.555 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1567e-05 2024-04-19T02:04:10.555 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.555 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.555 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:10.555 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194063+0000", 2024-04-19T02:04:10.555 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.1539999999999996e-06 2024-04-19T02:04:10.555 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.555 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.555 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.555 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194511+0000", 2024-04-19T02:04:10.556 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00044832999999999999 2024-04-19T02:04:10.556 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.556 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.556 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for blocked object", 2024-04-19T02:04:10.556 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194561+0000", 2024-04-19T02:04:10.556 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.9336999999999999e-05 2024-04-19T02:04:10.556 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.556 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.556 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.556 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.234276+0000", 2024-04-19T02:04:10.556 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.039715156000000001 2024-04-19T02:04:10.556 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.556 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.556 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:10.557 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.234405+0000", 2024-04-19T02:04:10.557 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00012947100000000001 2024-04-19T02:04:10.557 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.557 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.557 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.557 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.234507+0000", 2024-04-19T02:04:10.557 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000101234 2024-04-19T02:04:10.557 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.557 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.557 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.557 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.557 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.557 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:424 3.2s0 3:407ecba7:::smithi05930959-22:head [append 0~0] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:10.557 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194090+0000", 2024-04-19T02:04:10.557 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 40.331577394, 2024-04-19T02:04:10.558 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.04103888, 2024-04-19T02:04:10.558 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:10.558 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:10.558 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:10.558 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:10.558 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:10.558 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:10.558 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 424 2024-04-19T02:04:10.558 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.558 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:10.558 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.558 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:10.558 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194090+0000", 2024-04-19T02:04:10.559 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.559 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.559 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.559 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:10.559 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194090+0000", 2024-04-19T02:04:10.559 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.559 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.559 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.559 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:10.559 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194091+0000", 2024-04-19T02:04:10.559 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.9599999999999999e-07 2024-04-19T02:04:10.559 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.559 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.559 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:10.560 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194095+0000", 2024-04-19T02:04:10.560 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.019e-06 2024-04-19T02:04:10.560 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.560 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.560 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:10.560 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194104+0000", 2024-04-19T02:04:10.560 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.6030000000000005e-06 2024-04-19T02:04:10.560 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.560 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.560 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:10.560 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194110+0000", 2024-04-19T02:04:10.560 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.6880000000000002e-06 2024-04-19T02:04:10.560 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.560 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.561 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.561 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194597+0000", 2024-04-19T02:04:10.561 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00048649499999999998 2024-04-19T02:04:10.561 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.561 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.561 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:10.561 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194669+0000", 2024-04-19T02:04:10.561 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.2081000000000003e-05 2024-04-19T02:04:10.561 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.561 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.561 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:10.561 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195298+0000", 2024-04-19T02:04:10.561 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000628635 2024-04-19T02:04:10.561 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.561 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.562 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:10.562 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.201226+0000", 2024-04-19T02:04:10.562 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0059289199999999998 2024-04-19T02:04:10.562 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.562 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.562 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:10.562 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235058+0000", 2024-04-19T02:04:10.562 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.033831539000000001 2024-04-19T02:04:10.562 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.562 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.562 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.562 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235129+0000", 2024-04-19T02:04:10.562 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.1303999999999994e-05 2024-04-19T02:04:10.562 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.562 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.563 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.563 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.563 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.563 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:425 3.2s0 3:407ecba7:::smithi05930959-22:head [rollback a] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:10.563 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194118+0000", 2024-04-19T02:04:10.563 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 40.331549930999998, 2024-04-19T02:04:10.563 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.041424615999999997, 2024-04-19T02:04:10.563 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:10.563 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:10.563 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:10.563 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:10.563 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:10.563 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:10.563 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 425 2024-04-19T02:04:10.564 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.564 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:10.564 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.564 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:10.564 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194118+0000", 2024-04-19T02:04:10.564 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.564 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.564 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.564 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:10.564 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194118+0000", 2024-04-19T02:04:10.564 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.564 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.564 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.564 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:10.564 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194118+0000", 2024-04-19T02:04:10.565 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7799999999999997e-07 2024-04-19T02:04:10.565 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.565 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.565 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:10.565 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194120+0000", 2024-04-19T02:04:10.565 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9740000000000001e-06 2024-04-19T02:04:10.565 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.565 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.565 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:10.565 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194124+0000", 2024-04-19T02:04:10.565 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1540000000000004e-06 2024-04-19T02:04:10.565 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.565 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.565 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:10.566 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194127+0000", 2024-04-19T02:04:10.566 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1379999999999999e-06 2024-04-19T02:04:10.566 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.566 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.566 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.566 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195791+0000", 2024-04-19T02:04:10.566 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0016630779999999999 2024-04-19T02:04:10.566 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.566 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.566 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:10.566 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.195884+0000", 2024-04-19T02:04:10.566 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3885e-05 2024-04-19T02:04:10.566 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.566 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.567 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:10.567 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.196827+0000", 2024-04-19T02:04:10.567 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00094285400000000002 2024-04-19T02:04:10.567 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.567 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.567 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:10.567 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.201229+0000", 2024-04-19T02:04:10.567 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0044017359999999998 2024-04-19T02:04:10.567 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.567 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.567 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:10.567 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235479+0000", 2024-04-19T02:04:10.567 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.034249891999999997 2024-04-19T02:04:10.567 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.568 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.568 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.568 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235543+0000", 2024-04-19T02:04:10.568 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.3627000000000006e-05 2024-04-19T02:04:10.568 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.568 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.568 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.568 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.568 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.568 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:426 3.2s0 3:407ecba7:::smithi05930959-22:head [append 0~0] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:10.568 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194132+0000", 2024-04-19T02:04:10.568 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 40.331536319999998, 2024-04-19T02:04:10.568 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.041767809000000003, 2024-04-19T02:04:10.569 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:10.569 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:10.569 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:10.569 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:10.569 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:10.569 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:10.569 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 426 2024-04-19T02:04:10.569 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.569 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:10.569 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.569 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:10.569 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194132+0000", 2024-04-19T02:04:10.569 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.569 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.570 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.570 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:10.570 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194132+0000", 2024-04-19T02:04:10.570 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.570 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.570 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.570 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:10.570 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194132+0000", 2024-04-19T02:04:10.570 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.5199999999999998e-07 2024-04-19T02:04:10.570 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.570 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.570 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:10.570 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194133+0000", 2024-04-19T02:04:10.570 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.624e-06 2024-04-19T02:04:10.571 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.571 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.571 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:10.571 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194137+0000", 2024-04-19T02:04:10.571 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.3550000000000001e-06 2024-04-19T02:04:10.571 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.571 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.571 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:10.571 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194140+0000", 2024-04-19T02:04:10.571 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2229999999999999e-06 2024-04-19T02:04:10.571 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.571 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.571 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.571 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.198324+0000", 2024-04-19T02:04:10.571 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0041834799999999998 2024-04-19T02:04:10.572 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.572 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.572 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:10.572 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.198430+0000", 2024-04-19T02:04:10.572 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00010597100000000001 2024-04-19T02:04:10.572 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.572 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.572 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:10.572 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.199339+0000", 2024-04-19T02:04:10.572 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00090987100000000003 2024-04-19T02:04:10.572 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.572 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.572 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:10.572 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.201230+0000", 2024-04-19T02:04:10.573 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0018903170000000001 2024-04-19T02:04:10.573 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.573 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.573 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:10.573 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235836+0000", 2024-04-19T02:04:10.573 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.034606382999999998 2024-04-19T02:04:10.573 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.573 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.573 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.573 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235899+0000", 2024-04-19T02:04:10.573 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.3232999999999996e-05 2024-04-19T02:04:10.573 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.573 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.573 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.574 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.574 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.574 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:437 3.bs0 3:d6e3a590:::smithi05930959-43:head [copy-from ver 51] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:10.574 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194173+0000", 2024-04-19T02:04:10.574 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 40.331495074000003, 2024-04-19T02:04:10.574 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.054675901999999998, 2024-04-19T02:04:10.574 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:10.574 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:10.574 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:10.574 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:10.574 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:10.574 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:10.574 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 437 2024-04-19T02:04:10.575 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.575 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:10.575 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.575 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:10.575 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194173+0000", 2024-04-19T02:04:10.575 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.575 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.575 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.575 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:10.575 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194173+0000", 2024-04-19T02:04:10.575 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.575 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.575 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.575 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:10.575 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194173+0000", 2024-04-19T02:04:10.576 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.1100000000000002e-07 2024-04-19T02:04:10.576 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.576 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.576 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:10.576 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194179+0000", 2024-04-19T02:04:10.576 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.122e-06 2024-04-19T02:04:10.576 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.576 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.576 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:10.576 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194185+0000", 2024-04-19T02:04:10.576 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.3109999999999996e-06 2024-04-19T02:04:10.576 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.576 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.576 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:10.576 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194188+0000", 2024-04-19T02:04:10.577 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.0230000000000001e-06 2024-04-19T02:04:10.577 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.577 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.577 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.577 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194246+0000", 2024-04-19T02:04:10.577 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.8865e-05 2024-04-19T02:04:10.577 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.577 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.577 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:10.577 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194356+0000", 2024-04-19T02:04:10.577 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000109558 2024-04-19T02:04:10.577 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.577 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.577 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:10.577 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.240077+0000", 2024-04-19T02:04:10.577 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.045721108000000003 2024-04-19T02:04:10.578 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.578 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.578 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:10.578 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.244667+0000", 2024-04-19T02:04:10.578 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0045899000000000001 2024-04-19T02:04:10.578 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.578 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.578 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:10.578 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.248766+0000", 2024-04-19T02:04:10.578 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0040986579999999998 2024-04-19T02:04:10.578 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.578 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.578 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.578 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.248849+0000", 2024-04-19T02:04:10.578 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3046000000000004e-05 2024-04-19T02:04:10.579 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.581 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.581 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.581 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.581 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.581 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:438 3.bs0 3:d6e3a590:::smithi05930959-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:10.582 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194196+0000", 2024-04-19T02:04:10.582 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 40.331472261000002, 2024-04-19T02:04:10.582 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.047992689999999998, 2024-04-19T02:04:10.582 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:10.582 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:10.582 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:04:10.582 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:10.582 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:10.582 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:10.582 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 438 2024-04-19T02:04:10.582 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.582 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:10.582 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.582 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:10.583 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194196+0000", 2024-04-19T02:04:10.583 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.583 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.583 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.583 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:10.583 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194196+0000", 2024-04-19T02:04:10.583 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.583 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.583 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.583 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:10.583 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194196+0000", 2024-04-19T02:04:10.583 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.4700000000000002e-07 2024-04-19T02:04:10.583 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.583 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.584 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:10.584 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194198+0000", 2024-04-19T02:04:10.584 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.4320000000000002e-06 2024-04-19T02:04:10.584 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.584 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.584 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:10.584 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194204+0000", 2024-04-19T02:04:10.584 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.0850000000000002e-06 2024-04-19T02:04:10.584 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.584 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.584 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:10.584 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194209+0000", 2024-04-19T02:04:10.584 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.5090000000000004e-06 2024-04-19T02:04:10.584 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.585 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.585 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.585 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194533+0000", 2024-04-19T02:04:10.585 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00032412499999999998 2024-04-19T02:04:10.585 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.585 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.585 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "waiting for blocked object", 2024-04-19T02:04:10.585 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194593+0000", 2024-04-19T02:04:10.585 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.9854e-05 2024-04-19T02:04:10.585 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.585 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.585 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.585 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.242048+0000", 2024-04-19T02:04:10.585 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.047455152 2024-04-19T02:04:10.585 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.586 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.586 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:10.586 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.242113+0000", 2024-04-19T02:04:10.586 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.4573000000000007e-05 2024-04-19T02:04:10.586 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.586 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.586 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.586 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.242188+0000", 2024-04-19T02:04:10.586 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.5612999999999996e-05 2024-04-19T02:04:10.586 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.586 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.586 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.586 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.586 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.587 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:439 3.2s0 3:43ce42f3:::smithi05930959-27:head [append 0~0] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:10.587 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194231+0000", 2024-04-19T02:04:10.587 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 40.331436828000001, 2024-04-19T02:04:10.587 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.041997487, 2024-04-19T02:04:10.587 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:10.587 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:10.587 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:10.587 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:10.587 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:10.587 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:10.587 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 439 2024-04-19T02:04:10.587 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.587 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:10.588 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.588 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:10.588 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194231+0000", 2024-04-19T02:04:10.588 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.588 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.588 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.588 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:10.588 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194231+0000", 2024-04-19T02:04:10.588 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.588 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.588 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.588 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:10.588 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194232+0000", 2024-04-19T02:04:10.588 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7999999999999995e-07 2024-04-19T02:04:10.588 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.589 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.589 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:10.589 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194236+0000", 2024-04-19T02:04:10.589 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2139999999999998e-06 2024-04-19T02:04:10.589 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.589 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.589 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:10.589 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194243+0000", 2024-04-19T02:04:10.589 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.5979999999999999e-06 2024-04-19T02:04:10.589 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.589 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.589 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:10.589 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194248+0000", 2024-04-19T02:04:10.589 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.3370000000000001e-06 2024-04-19T02:04:10.590 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.590 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.590 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.590 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.204081+0000", 2024-04-19T02:04:10.590 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0098332009999999997 2024-04-19T02:04:10.590 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.590 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.590 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:10.590 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.204143+0000", 2024-04-19T02:04:10.590 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.156e-05 2024-04-19T02:04:10.590 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.590 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.590 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:10.590 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.204875+0000", 2024-04-19T02:04:10.591 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00073241099999999998 2024-04-19T02:04:10.591 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.591 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.591 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:10.591 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.208933+0000", 2024-04-19T02:04:10.591 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0040575990000000003 2024-04-19T02:04:10.591 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.591 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.591 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:10.591 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.236159+0000", 2024-04-19T02:04:10.591 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.027226274000000002 2024-04-19T02:04:10.591 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.591 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.591 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.591 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.236229+0000", 2024-04-19T02:04:10.592 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9713000000000002e-05 2024-04-19T02:04:10.592 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.592 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.592 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.592 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.592 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.592 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:440 3.2s0 3:43ce42f3:::smithi05930959-27:head [rollback a] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:10.592 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194253+0000", 2024-04-19T02:04:10.592 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 40.331414588000001, 2024-04-19T02:04:10.592 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.042233729999999997, 2024-04-19T02:04:10.592 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:10.592 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:10.592 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:10.592 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:10.593 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:10.593 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:10.593 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 440 2024-04-19T02:04:10.593 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.593 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:10.593 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.593 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:10.593 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194253+0000", 2024-04-19T02:04:10.593 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.593 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.593 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.593 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:10.593 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194253+0000", 2024-04-19T02:04:10.593 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.593 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.594 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.594 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:10.594 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194254+0000", 2024-04-19T02:04:10.594 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.8299999999999998e-07 2024-04-19T02:04:10.594 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.594 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.594 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:10.594 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194255+0000", 2024-04-19T02:04:10.594 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.7120000000000001e-06 2024-04-19T02:04:10.594 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.594 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.594 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:10.594 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194259+0000", 2024-04-19T02:04:10.594 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.517e-06 2024-04-19T02:04:10.594 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.595 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.595 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:10.595 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194261+0000", 2024-04-19T02:04:10.595 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.689e-06 2024-04-19T02:04:10.595 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.595 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.595 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.595 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.205980+0000", 2024-04-19T02:04:10.595 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.011718561000000001 2024-04-19T02:04:10.595 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.595 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.595 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:10.595 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.206074+0000", 2024-04-19T02:04:10.595 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.3556999999999999e-05 2024-04-19T02:04:10.596 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.596 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.596 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:10.596 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.206673+0000", 2024-04-19T02:04:10.596 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00059902100000000004 2024-04-19T02:04:10.596 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.596 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.596 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:10.596 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.208936+0000", 2024-04-19T02:04:10.596 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002262975 2024-04-19T02:04:10.596 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.596 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.596 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:10.596 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.236430+0000", 2024-04-19T02:04:10.597 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.027494338 2024-04-19T02:04:10.597 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.597 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.597 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.597 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.236487+0000", 2024-04-19T02:04:10.597 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.7077000000000003e-05 2024-04-19T02:04:10.597 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.597 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.597 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.597 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.597 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.597 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:441 3.2s0 3:43ce42f3:::smithi05930959-27:head [append 0~0] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:10.597 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.194270+0000", 2024-04-19T02:04:10.597 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 40.331398106000002, 2024-04-19T02:04:10.598 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.042541760999999997, 2024-04-19T02:04:10.598 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:10.598 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:10.598 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:10.598 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:10.598 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:10.598 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:10.598 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 441 2024-04-19T02:04:10.598 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.598 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:10.598 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.598 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:10.598 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194270+0000", 2024-04-19T02:04:10.598 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.599 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.599 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.599 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:10.599 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194270+0000", 2024-04-19T02:04:10.599 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.599 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.599 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.599 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:10.599 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194270+0000", 2024-04-19T02:04:10.599 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2500000000000001e-07 2024-04-19T02:04:10.599 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.599 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.599 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:10.599 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194271+0000", 2024-04-19T02:04:10.599 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3939999999999999e-06 2024-04-19T02:04:10.600 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.600 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.600 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:10.600 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194275+0000", 2024-04-19T02:04:10.600 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.827e-06 2024-04-19T02:04:10.600 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.600 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.600 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:10.600 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.194278+0000", 2024-04-19T02:04:10.600 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.7889999999999999e-06 2024-04-19T02:04:10.600 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.600 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.600 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.600 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.207160+0000", 2024-04-19T02:04:10.601 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.012881867999999999 2024-04-19T02:04:10.601 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.601 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.601 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:10.601 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.207262+0000", 2024-04-19T02:04:10.601 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000102142 2024-04-19T02:04:10.601 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.601 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.601 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:10.601 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.207787+0000", 2024-04-19T02:04:10.601 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00052537099999999995 2024-04-19T02:04:10.601 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.602 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.602 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:10.602 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.208937+0000", 2024-04-19T02:04:10.602 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00114908 2024-04-19T02:04:10.602 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.602 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.602 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:10.602 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.236712+0000", 2024-04-19T02:04:10.602 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.027775428000000001 2024-04-19T02:04:10.602 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.602 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.602 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.602 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.236811+0000", 2024-04-19T02:04:10.602 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9537000000000006e-05 2024-04-19T02:04:10.603 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.603 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.603 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.603 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.603 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.603 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(osd.7.18:2 3.2s0 3:465821b6:::smithi05930959-32:head [assert-version v14,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:10.603 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.199189+0000", 2024-04-19T02:04:10.603 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 40.326478870999999, 2024-04-19T02:04:10.603 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.035632812, 2024-04-19T02:04:10.603 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:10.603 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:10.603 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "started", 2024-04-19T02:04:10.603 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:10.603 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.604 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:10.604 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.199189+0000", 2024-04-19T02:04:10.604 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.604 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.604 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.604 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:10.604 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.199189+0000", 2024-04-19T02:04:10.604 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.604 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.604 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.604 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:10.604 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.199190+0000", 2024-04-19T02:04:10.604 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.5069999999999999e-06 2024-04-19T02:04:10.604 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.605 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.605 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:10.605 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.199198+0000", 2024-04-19T02:04:10.605 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.199e-06 2024-04-19T02:04:10.605 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.605 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.605 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:10.605 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.199211+0000", 2024-04-19T02:04:10.605 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3567000000000001e-05 2024-04-19T02:04:10.605 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.605 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.605 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:10.605 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.199220+0000", 2024-04-19T02:04:10.605 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2719999999999997e-06 2024-04-19T02:04:10.605 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.606 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.606 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.606 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.200485+0000", 2024-04-19T02:04:10.606 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.001265195 2024-04-19T02:04:10.606 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.606 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.606 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:10.606 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.200596+0000", 2024-04-19T02:04:10.606 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00011098 2024-04-19T02:04:10.606 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.606 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.606 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.606 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.234822+0000", 2024-04-19T02:04:10.606 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.034226092 2024-04-19T02:04:10.607 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.607 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.607 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.607 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.607 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.607 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpReadReply(3.2s0 46/26 ECSubReadReply(tid=166, attrs_read=0))", 2024-04-19T02:04:10.607 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.209587+0000", 2024-04-19T02:04:10.607 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 40.316080685999999, 2024-04-19T02:04:10.607 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.025249772, 2024-04-19T02:04:10.607 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:10.607 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:10.607 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:10.607 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:10.607 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.608 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:10.608 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.209587+0000", 2024-04-19T02:04:10.608 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.608 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.608 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.608 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:10.608 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.209587+0000", 2024-04-19T02:04:10.608 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.608 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.608 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.608 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:10.608 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.209588+0000", 2024-04-19T02:04:10.608 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 5.4099999999999999e-07 2024-04-19T02:04:10.608 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.609 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.609 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:10.609 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.209834+0000", 2024-04-19T02:04:10.609 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00024650200000000001 2024-04-19T02:04:10.609 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.609 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.609 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:10.609 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.209854+0000", 2024-04-19T02:04:10.609 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.9758999999999998e-05 2024-04-19T02:04:10.609 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.609 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.609 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:10.609 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.209867+0000", 2024-04-19T02:04:10.609 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2917e-05 2024-04-19T02:04:10.609 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.610 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.610 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.610 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.234535+0000", 2024-04-19T02:04:10.610 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.024668606999999999 2024-04-19T02:04:10.610 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.610 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.610 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.610 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.234837+0000", 2024-04-19T02:04:10.610 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00030144599999999997 2024-04-19T02:04:10.610 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.610 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.610 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.610 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.610 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.611 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 46/26 ECSubWriteReply(tid=164, last_complete=46'79, committed=1, applied=1))", 2024-04-19T02:04:10.611 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.210983+0000", 2024-04-19T02:04:10.611 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 40.314684878999998, 2024-04-19T02:04:10.611 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.024646257000000001, 2024-04-19T02:04:10.611 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:10.611 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:10.611 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:10.611 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:10.611 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.611 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:10.611 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210983+0000", 2024-04-19T02:04:10.611 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.611 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.611 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.611 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:10.612 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210983+0000", 2024-04-19T02:04:10.612 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.612 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.612 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.612 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:10.612 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210983+0000", 2024-04-19T02:04:10.612 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.2100000000000002e-07 2024-04-19T02:04:10.612 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.612 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.612 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:10.612 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210987+0000", 2024-04-19T02:04:10.612 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2789999999999999e-06 2024-04-19T02:04:10.612 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.612 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.612 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:10.612 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.210995+0000", 2024-04-19T02:04:10.613 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.3790000000000006e-06 2024-04-19T02:04:10.613 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.613 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.613 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:10.613 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.211007+0000", 2024-04-19T02:04:10.613 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.2373e-05 2024-04-19T02:04:10.613 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.613 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.613 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.613 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235615+0000", 2024-04-19T02:04:10.613 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.024607688999999999 2024-04-19T02:04:10.613 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.613 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.613 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.613 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235629+0000", 2024-04-19T02:04:10.613 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.4116e-05 2024-04-19T02:04:10.614 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.614 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.614 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.614 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.614 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.614 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 46/26 ECSubWriteReply(tid=167, last_complete=46'81, committed=1, applied=1))", 2024-04-19T02:04:10.614 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:03:30.211022+0000", 2024-04-19T02:04:10.614 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 40.314645994000003, 2024-04-19T02:04:10.614 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.024969574000000001, 2024-04-19T02:04:10.614 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:10.614 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:10.614 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "reached pg", 2024-04-19T02:04:10.614 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:10.614 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.614 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:10.615 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.211022+0000", 2024-04-19T02:04:10.615 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.615 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.615 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.615 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:10.615 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.211022+0000", 2024-04-19T02:04:10.615 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.615 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.615 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.615 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:10.615 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.211022+0000", 2024-04-19T02:04:10.615 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 3.2000000000000001e-07 2024-04-19T02:04:10.615 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.615 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.616 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:10.616 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.211025+0000", 2024-04-19T02:04:10.616 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 2.717e-06 2024-04-19T02:04:10.616 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.616 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.616 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:10.616 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.218661+0000", 2024-04-19T02:04:10.616 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0076366139999999999 2024-04-19T02:04:10.616 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.616 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.616 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:10.616 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.218669+0000", 2024-04-19T02:04:10.616 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 7.678e-06 2024-04-19T02:04:10.616 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.616 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.616 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.617 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235978+0000", 2024-04-19T02:04:10.617 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.017308430999999999 2024-04-19T02:04:10.617 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.617 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.617 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.617 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:03:30.235991+0000", 2024-04-19T02:04:10.617 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.3814e-05 2024-04-19T02:04:10.617 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.617 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.617 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.617 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.617 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.617 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:888 3.2s0 3:465821b6:::smithi05930959-32:head [rollback c] snapc 1d=[1d,1c,1b,19,17,16,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-04-19T02:04:10.617 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:04:00.154360+0000", 2024-04-19T02:04:10.617 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 10.371308162, 2024-04-19T02:04:10.618 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.027291403999999998, 2024-04-19T02:04:10.618 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:10.618 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:10.618 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:10.618 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:10.618 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:10.618 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:10.618 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 888 2024-04-19T02:04:10.618 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.618 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:10.618 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.618 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:10.618 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.154360+0000", 2024-04-19T02:04:10.618 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.619 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.619 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.619 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:10.619 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.154360+0000", 2024-04-19T02:04:10.619 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.619 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.619 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.619 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:10.619 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.154360+0000", 2024-04-19T02:04:10.619 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.0100000000000004e-07 2024-04-19T02:04:10.619 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.619 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.619 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:10.619 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.154365+0000", 2024-04-19T02:04:10.620 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.1330000000000001e-06 2024-04-19T02:04:10.620 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.620 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.620 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:10.620 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.154375+0000", 2024-04-19T02:04:10.620 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0652000000000001e-05 2024-04-19T02:04:10.620 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.620 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.620 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:10.620 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.154383+0000", 2024-04-19T02:04:10.620 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.1859999999999992e-06 2024-04-19T02:04:10.620 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.620 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.620 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.620 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.157163+0000", 2024-04-19T02:04:10.620 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0027800260000000001 2024-04-19T02:04:10.621 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.621 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.621 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:10.621 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.157263+0000", 2024-04-19T02:04:10.621 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 9.9890999999999996e-05 2024-04-19T02:04:10.621 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.621 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.621 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:10.621 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.157926+0000", 2024-04-19T02:04:10.621 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00066291400000000002 2024-04-19T02:04:10.621 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.621 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.621 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:10.621 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.160288+0000", 2024-04-19T02:04:10.621 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0023615440000000001 2024-04-19T02:04:10.622 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.622 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.622 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:10.622 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.181562+0000", 2024-04-19T02:04:10.622 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.021273888000000001 2024-04-19T02:04:10.622 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.622 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.622 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.622 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.181651+0000", 2024-04-19T02:04:10.622 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.9369000000000004e-05 2024-04-19T02:04:10.622 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.622 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.622 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.622 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.622 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.623 INFO:teuthology.orchestra.run.smithi059.stdout: "description": "osd_op(client.4509.0:889 3.2s0 3:465821b6:::smithi05930959-32:head [append 0~0] snapc 1d=[1d,1c,1b,19,17,16,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-04-19T02:04:10.623 INFO:teuthology.orchestra.run.smithi059.stdout: "initiated_at": "2024-04-19T02:04:00.154397+0000", 2024-04-19T02:04:10.623 INFO:teuthology.orchestra.run.smithi059.stdout: "age": 10.37127057, 2024-04-19T02:04:10.623 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.027438435000000001, 2024-04-19T02:04:10.623 INFO:teuthology.orchestra.run.smithi059.stdout: "continuous": false, 2024-04-19T02:04:10.623 INFO:teuthology.orchestra.run.smithi059.stdout: "type_data": { 2024-04-19T02:04:10.623 INFO:teuthology.orchestra.run.smithi059.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:10.623 INFO:teuthology.orchestra.run.smithi059.stdout: "client_info": { 2024-04-19T02:04:10.623 INFO:teuthology.orchestra.run.smithi059.stdout: "client": "client.4509", 2024-04-19T02:04:10.623 INFO:teuthology.orchestra.run.smithi059.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:10.623 INFO:teuthology.orchestra.run.smithi059.stdout: "tid": 889 2024-04-19T02:04:10.623 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.623 INFO:teuthology.orchestra.run.smithi059.stdout: "events": [ 2024-04-19T02:04:10.623 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.623 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "initiated", 2024-04-19T02:04:10.624 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.154397+0000", 2024-04-19T02:04:10.624 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.624 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.624 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.624 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", 2024-04-19T02:04:10.624 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.154397+0000", 2024-04-19T02:04:10.624 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0 2024-04-19T02:04:10.624 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.624 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.624 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", 2024-04-19T02:04:10.624 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.154398+0000", 2024-04-19T02:04:10.624 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.9400000000000005e-07 2024-04-19T02:04:10.624 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.624 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.624 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "all_read", 2024-04-19T02:04:10.624 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.154402+0000", 2024-04-19T02:04:10.625 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 4.4240000000000003e-06 2024-04-19T02:04:10.625 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.625 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.625 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "dispatched", 2024-04-19T02:04:10.625 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.154414+0000", 2024-04-19T02:04:10.625 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.1705999999999999e-05 2024-04-19T02:04:10.625 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.625 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.625 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", 2024-04-19T02:04:10.625 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.154422+0000", 2024-04-19T02:04:10.625 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 8.2369999999999992e-06 2024-04-19T02:04:10.625 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.625 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.625 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", 2024-04-19T02:04:10.625 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.158437+0000", 2024-04-19T02:04:10.626 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0040149929999999997 2024-04-19T02:04:10.626 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.626 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.626 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "started", 2024-04-19T02:04:10.626 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.158567+0000", 2024-04-19T02:04:10.626 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000130025 2024-04-19T02:04:10.626 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.626 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.626 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_started", 2024-04-19T02:04:10.626 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.159870+0000", 2024-04-19T02:04:10.626 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.0013025949999999999 2024-04-19T02:04:10.626 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.626 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.626 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "sub_op_committed", 2024-04-19T02:04:10.626 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.162624+0000", 2024-04-19T02:04:10.627 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.002754126 2024-04-19T02:04:10.627 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.627 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.627 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "commit_sent", 2024-04-19T02:04:10.627 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.181770+0000", 2024-04-19T02:04:10.627 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.019145549000000001 2024-04-19T02:04:10.627 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-04-19T02:04:10.627 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-04-19T02:04:10.627 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", 2024-04-19T02:04:10.627 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2024-04-19T02:04:00.181836+0000", 2024-04-19T02:04:10.627 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 6.6086000000000004e-05 2024-04-19T02:04:10.627 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.627 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.627 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.627 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-04-19T02:04:10.628 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-04-19T02:04:10.628 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:04:10.629 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-19T02:04:10.630 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:04:10.628+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:04:10.650 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-19T02:04:10.731 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:04:10.732+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:04:10.763 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-19T02:04:10.763 INFO:teuthology.orchestra.run.smithi081.stdout: "ops": [], 2024-04-19T02:04:10.764 INFO:teuthology.orchestra.run.smithi081.stdout: "num_ops": 0 2024-04-19T02:04:10.764 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-19T02:04:10.779 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-19T02:04:10.805 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-04-19T02:04:10.805 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T02:04:10.821 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-04-19T02:04:10.822 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T02:04:10.831 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:04:10.832+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:04:10.838 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-04-19T02:04:10.838 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T02:04:10.854 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-04-19T02:04:10.855 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T02:04:10.872 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-04-19T02:04:10.872 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T02:04:10.889 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-04-19T02:04:10.889 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T02:04:10.897 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:10.897 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:10.909 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-04-19T02:04:10.909 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T02:04:10.911 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-19T02:04:10.930 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-04-19T02:04:10.930 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T02:04:10.952 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-04-19T02:04:10.952 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T02:04:10.959 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:04:10.961+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:04:10.975 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-04-19T02:04:10.975 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T02:04:10.998 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-04-19T02:04:10.998 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T02:04:11.021 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-04-19T02:04:11.021 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T02:04:11.033 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:04:11.033+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:04:11.034 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:11.034 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:11.048 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-19T02:04:11.076 INFO:tasks.rados.rados.0.smithi059.stdout:388: done (5 left) 2024-04-19T02:04:11.076 INFO:tasks.rados.rados.0.smithi059.stdout:389: delete oid 11 current snap is 33 2024-04-19T02:04:11.076 INFO:tasks.rados.rados.0.smithi059.stdout:386: expect (ObjNum 1243376800 snap 21854 seq_num 1243869200) 2024-04-19T02:04:11.080 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-19T02:04:11.085 INFO:tasks.rados.rados.0.smithi059.stdout:383: finishing write tid 1 to smithi05930959-17 2024-04-19T02:04:11.086 INFO:tasks.rados.rados.0.smithi059.stdout:383: finishing write tid 2 to smithi05930959-17 2024-04-19T02:04:11.086 INFO:tasks.rados.rados.0.smithi059.stdout:383: finishing write tid 3 to smithi05930959-17 2024-04-19T02:04:11.086 INFO:tasks.rados.rados.0.smithi059.stdout:383: oid 17 updating version 0 to 202 2024-04-19T02:04:11.086 INFO:tasks.rados.rados.0.smithi059.stdout:383: oid 17 version 202 is already newer than 201 2024-04-19T02:04:11.086 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 17 v 202 (ObjNum 165 snap 33 seq_num 165) dirty exists 2024-04-19T02:04:11.086 INFO:tasks.rados.rados.0.smithi059.stdout:383: left oid 17 (ObjNum 165 snap 33 seq_num 165) 2024-04-19T02:04:11.086 INFO:tasks.rados.rados.0.smithi059.stdout:387: finishing write tid 1 to smithi05930959-35 2024-04-19T02:04:11.086 INFO:tasks.rados.rados.0.smithi059.stdout:387: finishing write tid 2 to smithi05930959-35 2024-04-19T02:04:11.086 INFO:tasks.rados.rados.0.smithi059.stdout:387: finishing write tid 3 to smithi05930959-35 2024-04-19T02:04:11.086 INFO:tasks.rados.rados.0.smithi059.stdout:387: oid 35 updating version 0 to 106 2024-04-19T02:04:11.086 INFO:tasks.rados.rados.0.smithi059.stdout:387: oid 35 updating version 106 to 107 2024-04-19T02:04:11.086 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 35 v 107 (ObjNum 166 snap 33 seq_num 166) dirty exists 2024-04-19T02:04:11.087 INFO:tasks.rados.rados.0.smithi059.stdout:387: left oid 35 (ObjNum 166 snap 33 seq_num 166) 2024-04-19T02:04:11.087 INFO:tasks.rados.rados.0.smithi059.stdout:385: expect (ObjNum 142 snap 24 seq_num 142) 2024-04-19T02:04:11.134 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:04:11.136+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:04:11.220 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:11.220 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:11.235 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-19T02:04:11.235 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:04:11.236+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:04:11.269 INFO:tasks.rados.rados.0.smithi059.stdout:384: finishing copy_from racing read to smithi05930959-3 2024-04-19T02:04:11.269 INFO:tasks.rados.rados.0.smithi059.stdout:384: finishing copy_from to smithi05930959-3 2024-04-19T02:04:11.269 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 3 v 27 (ObjNum 150 snap 26 seq_num 150) dirty exists 2024-04-19T02:04:11.269 INFO:tasks.rados.rados.0.smithi059.stdout:383: done (5 left) 2024-04-19T02:04:11.269 INFO:tasks.rados.rados.0.smithi059.stdout:384: done (4 left) 2024-04-19T02:04:11.269 INFO:tasks.rados.rados.0.smithi059.stdout:385: done (3 left) 2024-04-19T02:04:11.269 INFO:tasks.rados.rados.0.smithi059.stdout:386: done (2 left) 2024-04-19T02:04:11.270 INFO:tasks.rados.rados.0.smithi059.stdout:387: done (1 left) 2024-04-19T02:04:11.270 INFO:tasks.rados.rados.0.smithi059.stdout:389: done (0 left) 2024-04-19T02:04:11.270 INFO:tasks.rados.rados.0.smithi059.stdout:390: read oid 32 snap -1 2024-04-19T02:04:11.270 INFO:tasks.rados.rados.0.smithi059.stdout:390: expect (ObjNum 31 snap 0 seq_num 31) 2024-04-19T02:04:11.270 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 6 current snap is 33 2024-04-19T02:04:11.270 INFO:tasks.rados.rados.0.smithi059.stdout:391: seq_num 168 ranges {2621440=425984} 2024-04-19T02:04:11.305 INFO:tasks.rados.rados.0.smithi059.stdout:391: writing smithi05930959-6 from 2621440 to 3047424 tid 1 2024-04-19T02:04:11.305 INFO:tasks.rados.rados.0.smithi059.stdout:392: setattr oid 17 current snap is 33 2024-04-19T02:04:11.305 INFO:tasks.rados.rados.0.smithi059.stdout:390: expect (ObjNum 31 snap 0 seq_num 31) 2024-04-19T02:04:11.331 INFO:tasks.rados.rados.0.smithi059.stdout:391: finishing write tid 1 to smithi05930959-6 2024-04-19T02:04:11.331 INFO:tasks.rados.rados.0.smithi059.stdout:391: finishing write tid 2 to smithi05930959-6 2024-04-19T02:04:11.332 INFO:tasks.rados.rados.0.smithi059.stdout:391: finishing write tid 3 to smithi05930959-6 2024-04-19T02:04:11.332 INFO:tasks.rados.rados.0.smithi059.stdout:391: oid 6 updating version 0 to 97 2024-04-19T02:04:11.332 INFO:tasks.rados.rados.0.smithi059.stdout:391: oid 6 updating version 97 to 98 2024-04-19T02:04:11.332 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 6 v 98 (ObjNum 168 snap 33 seq_num 168) dirty exists 2024-04-19T02:04:11.332 INFO:tasks.rados.rados.0.smithi059.stdout:391: left oid 6 (ObjNum 168 snap 33 seq_num 168) 2024-04-19T02:04:11.332 INFO:tasks.rados.rados.0.smithi059.stdout:390: done (2 left) 2024-04-19T02:04:11.332 INFO:tasks.rados.rados.0.smithi059.stdout:391: done (1 left) 2024-04-19T02:04:11.332 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 22 current snap is 33 2024-04-19T02:04:11.332 INFO:tasks.rados.rados.0.smithi059.stdout:393: seq_num 169 ranges {360448=442368} 2024-04-19T02:04:11.336 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:04:11.337+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:04:11.342 INFO:tasks.rados.rados.0.smithi059.stdout:393: writing smithi05930959-22 from 360448 to 802816 tid 1 2024-04-19T02:04:11.342 INFO:tasks.rados.rados.0.smithi059.stdout:394: rollback oid 3 current snap is 33 2024-04-19T02:04:11.342 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 3 to 28 2024-04-19T02:04:11.342 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 17 v 209 (ObjNum 165 snap 33 seq_num 165) dirty exists 2024-04-19T02:04:11.342 INFO:tasks.rados.rados.0.smithi059.stdout:392: done (2 left) 2024-04-19T02:04:11.342 INFO:tasks.rados.rados.0.smithi059.stdout:395: rollback oid 8 current snap is 33 2024-04-19T02:04:11.342 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 8 to 14 2024-04-19T02:04:11.342 INFO:tasks.rados.rados.0.smithi059.stdout:396: rollback oid 35 current snap is 33 2024-04-19T02:04:11.342 INFO:tasks.rados.rados.0.smithi059.stdout:rollback oid 35 to 28 2024-04-19T02:04:11.342 INFO:tasks.rados.rados.0.smithi059.stdout:append oid 42 current snap is 33 2024-04-19T02:04:11.342 INFO:tasks.rados.rados.0.smithi059.stdout:397: seq_num 170 ranges {3145728=458752} 2024-04-19T02:04:11.368 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:11.368 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:11.381 INFO:tasks.rados.rados.0.smithi059.stdout:397: writing smithi05930959-42 from 3145728 to 3604480 tid 1 2024-04-19T02:04:11.381 INFO:tasks.rados.rados.0.smithi059.stdout:398: snap_create 2024-04-19T02:04:11.381 INFO:tasks.rados.rados.0.smithi059.stdout:395: finishing rollback tid 1 to smithi05930959-8 2024-04-19T02:04:11.381 INFO:tasks.rados.rados.0.smithi059.stdout:394: finishing rollback tid 0 to smithi05930959-3 2024-04-19T02:04:11.382 INFO:tasks.rados.rados.0.smithi059.stdout:395: finishing rollback tid 2 to smithi05930959-8 2024-04-19T02:04:11.382 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 8 v 87 (ObjNum 57 snap 1 seq_num 57) dirty exists 2024-04-19T02:04:11.382 INFO:tasks.rados.rados.0.smithi059.stdout:394: finishing rollback tid 1 to smithi05930959-3 2024-04-19T02:04:11.382 INFO:tasks.rados.rados.0.smithi059.stdout:394: finishing rollback tid 2 to smithi05930959-3 2024-04-19T02:04:11.382 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 3 v 147 (ObjNum 108 snap 15 seq_num 108) dirty exists 2024-04-19T02:04:11.382 INFO:tasks.rados.rados.0.smithi059.stdout:396: finishing rollback tid 0 to smithi05930959-35 2024-04-19T02:04:11.382 INFO:tasks.rados.rados.0.smithi059.stdout:393: finishing write tid 1 to smithi05930959-22 2024-04-19T02:04:11.382 INFO:tasks.rados.rados.0.smithi059.stdout:393: finishing write tid 2 to smithi05930959-22 2024-04-19T02:04:11.382 INFO:tasks.rados.rados.0.smithi059.stdout:396: finishing rollback tid 1 to smithi05930959-35 2024-04-19T02:04:11.382 INFO:tasks.rados.rados.0.smithi059.stdout:396: finishing rollback tid 2 to smithi05930959-35 2024-04-19T02:04:11.382 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 35 v 114 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-19T02:04:11.382 INFO:tasks.rados.rados.0.smithi059.stdout:393: finishing write tid 3 to smithi05930959-22 2024-04-19T02:04:11.382 INFO:tasks.rados.rados.0.smithi059.stdout:393: oid 22 updating version 0 to 212 2024-04-19T02:04:11.382 INFO:tasks.rados.rados.0.smithi059.stdout:393: oid 22 version 212 is already newer than 211 2024-04-19T02:04:11.383 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 22 v 212 (ObjNum 169 snap 33 seq_num 169) dirty exists 2024-04-19T02:04:11.383 INFO:tasks.rados.rados.0.smithi059.stdout:393: left oid 22 (ObjNum 169 snap 33 seq_num 169) 2024-04-19T02:04:11.384 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-19T02:04:11.389 INFO:tasks.rados.rados.0.smithi059.stdout:397: finishing write tid 1 to smithi05930959-42 2024-04-19T02:04:11.390 INFO:tasks.rados.rados.0.smithi059.stdout:397: finishing write tid 2 to smithi05930959-42 2024-04-19T02:04:11.394 INFO:tasks.rados.rados.0.smithi059.stdout:397: finishing write tid 3 to smithi05930959-42 2024-04-19T02:04:11.394 INFO:tasks.rados.rados.0.smithi059.stdout:397: oid 42 updating version 0 to 100 2024-04-19T02:04:11.394 INFO:tasks.rados.rados.0.smithi059.stdout:397: oid 42 updating version 100 to 101 2024-04-19T02:04:11.394 INFO:tasks.rados.rados.0.smithi059.stdout:update_object_version oid 42 v 101 (ObjNum 170 snap 33 seq_num 170) dirty exists 2024-04-19T02:04:11.394 INFO:tasks.rados.rados.0.smithi059.stdout:397: left oid 42 (ObjNum 170 snap 33 seq_num 170) 2024-04-19T02:04:11.437 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:04:11.436+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:04:11.538 INFO:tasks.ceph.osd.11.smithi179.stderr:2024-04-19T02:04:11.540+0000 7f954e787640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25311) UID: 0 2024-04-19T02:04:11.549 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:11.549 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:11.563 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-19T02:04:11.639 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:04:11.637+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:04:11.685 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:11.686 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:11.699 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-19T02:04:11.739 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:04:11.741+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:04:11.840 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:04:11.841+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:04:11.872 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:11.872 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:11.886 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-19T02:04:11.941 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:04:11.941+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:04:12.007 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:12.007 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:12.021 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-19T02:04:12.042 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:04:12.044+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:04:12.053 INFO:teuthology.orchestra.run.smithi059.stderr:nodeep-scrub is set 2024-04-19T02:04:12.057 INFO:tasks.rados.rados.0.smithi059.stdout:393: done (5 left) 2024-04-19T02:04:12.057 INFO:tasks.rados.rados.0.smithi059.stdout:394: done (4 left) 2024-04-19T02:04:12.057 INFO:tasks.rados.rados.0.smithi059.stdout:395: done (3 left) 2024-04-19T02:04:12.057 INFO:tasks.rados.rados.0.smithi059.stdout:396: done (2 left) 2024-04-19T02:04:12.057 INFO:tasks.rados.rados.0.smithi059.stdout:397: done (1 left) 2024-04-19T02:04:12.057 INFO:tasks.rados.rados.0.smithi059.stdout:398: done (0 left) 2024-04-19T02:04:12.057 INFO:tasks.rados.rados.0.smithi059.stdout:399: read oid 33 snap -1 2024-04-19T02:04:12.057 INFO:tasks.rados.rados.0.smithi059.stdout:399: expect deleted 2024-04-19T02:04:12.057 INFO:tasks.rados.rados.0.smithi059.stdout: waiting on 1 2024-04-19T02:04:12.061 INFO:tasks.rados.rados.0.smithi059.stdout:399: done (0 left) 2024-04-19T02:04:12.069 INFO:tasks.rados.rados.0.smithi059.stderr:0 errors. 2024-04-19T02:04:12.069 INFO:tasks.rados.rados.0.smithi059.stderr: 2024-04-19T02:04:12.078 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-19T02:04:12.141 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:12.141 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:12.143 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-04-19T02:04:12.145+0000 7f1f92f51640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25041) UID: 0 2024-04-19T02:04:12.155 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-19T02:04:12.244 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:04:12.245+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:04:12.322 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:12.322 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:12.336 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-19T02:04:12.346 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:04:12.348+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:04:12.434 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:04:12.434 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":88,"fsid":"db9b8c36-b038-40a6-b749-a60ca7443e45","created":"2024-04-19T02:02:47.352143+0000","modified":"2024-04-19T02:04:12.052562+0000","last_up_change":"2024-04-19T02:02:59.513773+0000","last_in_change":"2024-04-19T02:02:51.039271+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-19T02:02:58.287078+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.92051815986633301,"average_primary_affinity":0.92000001668930054,"average_primary_affinity_weighted":0.92000001668930054}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-19T02:03:00.613285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.5199999809265137,"score_stable":4.5199999809265137,"optimal_score":0.82999998331069946,"raw_score_acting":3.75,"raw_score_stable":3.75,"primary_affinity_weighted":0.92051815986633301,"average_primary_affinity":0.92000001668930054,"average_primary_affinity_weighted":0.92000001668930054}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2024-04-19T02:03:09.633037+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":8,"min_size":5,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":35,"snap_epoch":88,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lrcprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":16384,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"dff2e9d5-c79f-4cc8-b68c-d1d35cc3dfeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1145933874},{"type":"v1","addr":"172.21.15.59:6809","nonce":1145933874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1145933874},{"type":"v1","addr":"172.21.15.59:6811","nonce":1145933874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1145933874},{"type":"v1","addr":"172.21.15.59:6815","nonce":1145933874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1145933874},{"type":"v1","addr":"172.21.15.59:6813","nonce":1145933874}]},"public_addr":"172.21.15.59:6809/1145933874","cluster_addr":"172.21.15.59:6811/1145933874","heartbeat_back_addr":"172.21.15.59:6815/1145933874","heartbeat_front_addr":"172.21.15.59:6813/1145933874","state":["exists","up"]},{"osd":1,"uuid":"30cb7aa1-0bf8-4bc8-8bc5-f8b59870624e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":3807848039},{"type":"v1","addr":"172.21.15.59:6801","nonce":3807848039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":3807848039},{"type":"v1","addr":"172.21.15.59:6803","nonce":3807848039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":3807848039},{"type":"v1","addr":"172.21.15.59:6807","nonce":3807848039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":3807848039},{"type":"v1","addr":"172.21.15.59:6805","nonce":3807848039}]},"public_addr":"172.21.15.59:6801/3807848039","cluster_addr":"172.21.15.59:6803/3807848039","heartbeat_back_addr":"172.21.15.59:6807/3807848039","heartbeat_front_addr":"172.21.15.59:6805/3807848039","state":["exists","up"]},{"osd":2,"uuid":"2e8fca1b-e008-4e58-865e-09a658f6edd9","up":1,"in":1,"weight":1,"primary_affinity":0.0462188720703125,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":420007638},{"type":"v1","addr":"172.21.15.59:6817","nonce":420007638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":420007638},{"type":"v1","addr":"172.21.15.59:6822","nonce":420007638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6829","nonce":420007638},{"type":"v1","addr":"172.21.15.59:6831","nonce":420007638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":420007638},{"type":"v1","addr":"172.21.15.59:6827","nonce":420007638}]},"public_addr":"172.21.15.59:6817/420007638","cluster_addr":"172.21.15.59:6822/420007638","heartbeat_back_addr":"172.21.15.59:6831/420007638","heartbeat_front_addr":"172.21.15.59:6827/420007638","state":["exists","up"]},{"osd":3,"uuid":"32289c9a-b7d9-4bcd-9f66-fa68f0c9796b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1791539182},{"type":"v1","addr":"172.21.15.59:6819","nonce":1791539182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6821","nonce":1791539182},{"type":"v1","addr":"172.21.15.59:6823","nonce":1791539182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1791539182},{"type":"v1","addr":"172.21.15.59:6830","nonce":1791539182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6825","nonce":1791539182},{"type":"v1","addr":"172.21.15.59:6826","nonce":1791539182}]},"public_addr":"172.21.15.59:6819/1791539182","cluster_addr":"172.21.15.59:6823/1791539182","heartbeat_back_addr":"172.21.15.59:6830/1791539182","heartbeat_front_addr":"172.21.15.59:6826/1791539182","state":["exists","up"]},{"osd":4,"uuid":"844daeea-860b-4853-a759-994531e5de3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":2921507000},{"type":"v1","addr":"172.21.15.81:6808","nonce":2921507000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":2921507000},{"type":"v1","addr":"172.21.15.81:6811","nonce":2921507000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":2921507000},{"type":"v1","addr":"172.21.15.81:6815","nonce":2921507000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2921507000},{"type":"v1","addr":"172.21.15.81:6813","nonce":2921507000}]},"public_addr":"172.21.15.81:6808/2921507000","cluster_addr":"172.21.15.81:6811/2921507000","heartbeat_back_addr":"172.21.15.81:6815/2921507000","heartbeat_front_addr":"172.21.15.81:6813/2921507000","state":["exists","up"]},{"osd":5,"uuid":"f43ad412-74bb-43a6-95e7-25ee06bde025","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3925467179},{"type":"v1","addr":"172.21.15.81:6818","nonce":3925467179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":3925467179},{"type":"v1","addr":"172.21.15.81:6821","nonce":3925467179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":3925467179},{"type":"v1","addr":"172.21.15.81:6826","nonce":3925467179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":3925467179},{"type":"v1","addr":"172.21.15.81:6824","nonce":3925467179}]},"public_addr":"172.21.15.81:6818/3925467179","cluster_addr":"172.21.15.81:6821/3925467179","heartbeat_back_addr":"172.21.15.81:6826/3925467179","heartbeat_front_addr":"172.21.15.81:6824/3925467179","state":["exists","up"]},{"osd":6,"uuid":"07ccbc36-1f5d-4595-8578-09362b5e212f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3714790214},{"type":"v1","addr":"172.21.15.81:6801","nonce":3714790214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3714790214},{"type":"v1","addr":"172.21.15.81:6803","nonce":3714790214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6807","nonce":3714790214},{"type":"v1","addr":"172.21.15.81:6809","nonce":3714790214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3714790214},{"type":"v1","addr":"172.21.15.81:6805","nonce":3714790214}]},"public_addr":"172.21.15.81:6801/3714790214","cluster_addr":"172.21.15.81:6803/3714790214","heartbeat_back_addr":"172.21.15.81:6809/3714790214","heartbeat_front_addr":"172.21.15.81:6805/3714790214","state":["exists","up"]},{"osd":7,"uuid":"6b2247e7-ebf8-4c0e-89ae-e15985f59aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":2170932617},{"type":"v1","addr":"172.21.15.81:6820","nonce":2170932617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":2170932617},{"type":"v1","addr":"172.21.15.81:6827","nonce":2170932617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":2170932617},{"type":"v1","addr":"172.21.15.81:6831","nonce":2170932617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":2170932617},{"type":"v1","addr":"172.21.15.81:6829","nonce":2170932617}]},"public_addr":"172.21.15.81:6820/2170932617","cluster_addr":"172.21.15.81:6827/2170932617","heartbeat_back_addr":"172.21.15.81:6831/2170932617","heartbeat_front_addr":"172.21.15.81:6829/2170932617","state":["exists","up"]},{"osd":8,"uuid":"7142efd0-2144-4973-a01d-69a064b9b51f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2053612068},{"type":"v1","addr":"172.21.15.179:6825","nonce":2053612068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2053612068},{"type":"v1","addr":"172.21.15.179:6827","nonce":2053612068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2053612068},{"type":"v1","addr":"172.21.15.179:6831","nonce":2053612068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2053612068},{"type":"v1","addr":"172.21.15.179:6829","nonce":2053612068}]},"public_addr":"172.21.15.179:6825/2053612068","cluster_addr":"172.21.15.179:6827/2053612068","heartbeat_back_addr":"172.21.15.179:6831/2053612068","heartbeat_front_addr":"172.21.15.179:6829/2053612068","state":["exists","up"]},{"osd":9,"uuid":"30a29baa-53cc-4c29-ab41-af14ead5c539","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2641050728},{"type":"v1","addr":"172.21.15.179:6801","nonce":2641050728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2641050728},{"type":"v1","addr":"172.21.15.179:6803","nonce":2641050728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2641050728},{"type":"v1","addr":"172.21.15.179:6807","nonce":2641050728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2641050728},{"type":"v1","addr":"172.21.15.179:6805","nonce":2641050728}]},"public_addr":"172.21.15.179:6801/2641050728","cluster_addr":"172.21.15.179:6803/2641050728","heartbeat_back_addr":"172.21.15.179:6807/2641050728","heartbeat_front_addr":"172.21.15.179:6805/2641050728","state":["exists","up"]},{"osd":10,"uuid":"fae84343-8b7a-4bd3-aeef-823f2f6058e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3465727116},{"type":"v1","addr":"172.21.15.179:6817","nonce":3465727116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3465727116},{"type":"v1","addr":"172.21.15.179:6819","nonce":3465727116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3465727116},{"type":"v1","addr":"172.21.15.179:6823","nonce":3465727116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3465727116},{"type":"v1","addr":"172.21.15.179:6821","nonce":3465727116}]},"public_addr":"172.21.15.179:6817/3465727116","cluster_addr":"172.21.15.179:6819/3465727116","heartbeat_back_addr":"172.21.15.179:6823/3465727116","heartbeat_front_addr":"172.21.15.179:6821/3465727116","state":["exists","up"]},{"osd":11,"uuid":"35858134-6864-4da1-88c2-4b48166febab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":4047209718},{"type":"v1","addr":"172.21.15.179:6809","nonce":4047209718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":4047209718},{"type":"v1","addr":"172.21.15.179:6811","nonce":4047209718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4047209718},{"type":"v1","addr":"172.21.15.179:6815","nonce":4047209718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4047209718},{"type":"v1","addr":"172.21.15.179:6813","nonce":4047209718}]},"public_addr":"172.21.15.179:6809/4047209718","cluster_addr":"172.21.15.179:6811/4047209718","heartbeat_back_addr":"172.21.15.179:6815/4047209718","heartbeat_front_addr":"172.21.15.179:6813/4047209718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.527995+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.434032+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.570606+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.621023+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.511399+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.404135+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.624821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.705799+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.547852+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.272469+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:02:56.401570+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:03:57.837672+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lrcprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","l":"3","m":"2","name":"lrcprofile","plugin":"lrc"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":27,"length":1},{"begin":29,"length":1},{"begin":33,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":18,"length":1},{"begin":22,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T02:04:12.446 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:04:12.445+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:04:12.454 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T02:04:12.455 INFO:teuthology.orchestra.run.smithi081.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:12.455 INFO:teuthology.orchestra.run.smithi081.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:12.468 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-19T02:04:12.548 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:04:12.549+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:04:12.599 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:12.599 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:12.613 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-19T02:04:12.649 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-04-19T02:04:12.648+0000 7f85c791a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25037) UID: 0 2024-04-19T02:04:12.732 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:12.732 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:12.746 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-19T02:04:12.749 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:04:12.749+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:04:12.795 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:04:12.796 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-19T02:04:12.813 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":114,"stamp":"2024-04-19T02:04:12.224138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":127009312,"num_objects":110,"num_object_clones":61,"num_object_copies":868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":110,"num_whiteouts":7,"num_read":567,"num_read_kb":238566,"num_write":967,"num_write_kb":162442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1334,"ondisk_log_size":1334,"up":146,"acting":146,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1132462080,"kb_used":737380,"kb_used_data":259056,"kb_used_omap":18,"kb_used_meta":478125,"kb_avail":1131724700,"statfs":{"total":1159641169920,"available":1158886092800,"internally_reserved":0,"allocated":265273344,"data_stored":436596742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":489600313},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":884736,"num_objects":3,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":-1,"num_read":3,"num_read_kb":3,"num_write":8,"num_write_kb":864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001138"},"pg_stats":[{"pgid":"3.f","version":"82'54","reported_seq":348,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-19T02:04:08.061219+0000","last_change":"2024-04-19T02:04:06.185537+0000","last_active":"2024-04-19T02:04:08.061219+0000","last_peered":"2024-04-19T02:04:08.061219+0000","last_clean":"2024-04-19T02:04:08.061219+0000","last_became_active":"2024-04-19T02:03:11.728136+0000","last_became_peered":"2024-04-19T02:03:11.728136+0000","last_unstale":"2024-04-19T02:04:08.061219+0000","last_undegraded":"2024-04-19T02:04:08.061219+0000","last_fullsized":"2024-04-19T02:04:08.061219+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:21.479575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.021741834000000002,"stat_sum":{"num_bytes":8323072,"num_objects":5,"num_object_clones":2,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23677,"num_write":29,"num_write_kb":9136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,2,3,5,8,0,7],"acting":[11,4,2,3,5,8,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"},{"start":"16","length":"1"},{"start":"1d","length":"1"}]},{"pgid":"3.e","version":"80'60","reported_seq":349,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-19T02:04:08.085464+0000","last_change":"2024-04-19T02:04:06.162308+0000","last_active":"2024-04-19T02:04:08.085464+0000","last_peered":"2024-04-19T02:04:08.085464+0000","last_clean":"2024-04-19T02:04:08.085464+0000","last_became_active":"2024-04-19T02:03:11.730135+0000","last_became_peered":"2024-04-19T02:03:11.730135+0000","last_unstale":"2024-04-19T02:04:08.085464+0000","last_undegraded":"2024-04-19T02:04:08.085464+0000","last_fullsized":"2024-04-19T02:04:08.085464+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:10.658227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069123299999999997,"stat_sum":{"num_bytes":6586368,"num_objects":5,"num_object_clones":2,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":7016,"num_write":27,"num_write_kb":7648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5,11,0,2,3,9],"acting":[7,4,5,11,0,2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"},{"start":"16","length":"1"},{"start":"1d","length":"1"}]},{"pgid":"3.d","version":"82'112","reported_seq":403,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-19T02:04:09.070624+0000","last_change":"2024-04-19T02:04:06.204236+0000","last_active":"2024-04-19T02:04:09.070624+0000","last_peered":"2024-04-19T02:04:09.070624+0000","last_clean":"2024-04-19T02:04:09.070624+0000","last_became_active":"2024-04-19T02:03:11.632682+0000","last_became_peered":"2024-04-19T02:03:11.632682+0000","last_unstale":"2024-04-19T02:04:09.070624+0000","last_undegraded":"2024-04-19T02:04:09.070624+0000","last_fullsized":"2024-04-19T02:04:09.070624+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:27.723042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032276640000000002,"stat_sum":{"num_bytes":12435456,"num_objects":10,"num_object_clones":6,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":35,"num_read_kb":17086,"num_write":63,"num_write_kb":14640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7,4,5,3,9,0],"acting":[8,11,7,4,5,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"},{"start":"16","length":"1"},{"start":"1d","length":"1"}]},{"pgid":"3.c","version":"82'89","reported_seq":397,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-19T02:04:08.091810+0000","last_change":"2024-04-19T02:04:06.207905+0000","last_active":"2024-04-19T02:04:08.091810+0000","last_peered":"2024-04-19T02:04:08.091810+0000","last_clean":"2024-04-19T02:04:08.091810+0000","last_became_active":"2024-04-19T02:03:11.731568+0000","last_became_peered":"2024-04-19T02:03:11.731568+0000","last_unstale":"2024-04-19T02:04:08.091810+0000","last_undegraded":"2024-04-19T02:04:08.091810+0000","last_fullsized":"2024-04-19T02:04:08.091810+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:34.630411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033384486999999997,"stat_sum":{"num_bytes":9256960,"num_objects":8,"num_object_clones":4,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":57,"num_read_kb":34628,"num_write":39,"num_write_kb":13168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,11,7,4,2,0,5],"acting":[9,3,11,7,4,2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"},{"start":"16","length":"1"},{"start":"1d","length":"1"}]},{"pgid":"3.b","version":"82'81","reported_seq":372,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-19T02:04:08.091856+0000","last_change":"2024-04-19T02:04:06.185034+0000","last_active":"2024-04-19T02:04:08.091856+0000","last_peered":"2024-04-19T02:04:08.091856+0000","last_clean":"2024-04-19T02:04:08.091856+0000","last_became_active":"2024-04-19T02:03:11.677316+0000","last_became_peered":"2024-04-19T02:03:11.677316+0000","last_unstale":"2024-04-19T02:04:08.091856+0000","last_undegraded":"2024-04-19T02:04:08.091856+0000","last_fullsized":"2024-04-19T02:04:08.091856+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:12.843307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022408922000000001,"stat_sum":{"num_bytes":8175616,"num_objects":7,"num_object_clones":4,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":38,"num_read_kb":13523,"num_write":53,"num_write_kb":12272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8,6,4,10,1,7],"acting":[3,0,8,6,4,10,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"},{"start":"16","length":"1"},{"start":"1d","length":"1"}]},{"pgid":"3.a","version":"82'104","reported_seq":398,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-19T02:04:08.061294+0000","last_change":"2024-04-19T02:04:06.199898+0000","last_active":"2024-04-19T02:04:08.061294+0000","last_peered":"2024-04-19T02:04:08.061294+0000","last_clean":"2024-04-19T02:04:08.061294+0000","last_became_active":"2024-04-19T02:03:11.727996+0000","last_became_peered":"2024-04-19T02:03:11.727996+0000","last_unstale":"2024-04-19T02:04:08.061294+0000","last_undegraded":"2024-04-19T02:04:08.061294+0000","last_fullsized":"2024-04-19T02:04:08.061294+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:24.114301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.035661384999999997,"stat_sum":{"num_bytes":17711104,"num_objects":9,"num_object_clones":5,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":46,"num_read_kb":50603,"num_write":93,"num_write_kb":17296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,1,9,5,7,3,2],"acting":[11,8,1,9,5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"},{"start":"16","length":"1"},{"start":"1d","length":"1"}]},{"pgid":"3.9","version":"84'66","reported_seq":357,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-19T02:04:09.070623+0000","last_change":"2024-04-19T02:04:06.193378+0000","last_active":"2024-04-19T02:04:09.070623+0000","last_peered":"2024-04-19T02:04:09.070623+0000","last_clean":"2024-04-19T02:04:09.070623+0000","last_became_active":"2024-04-19T02:03:11.729255+0000","last_became_peered":"2024-04-19T02:03:11.729255+0000","last_unstale":"2024-04-19T02:04:09.070623+0000","last_undegraded":"2024-04-19T02:04:09.070623+0000","last_fullsized":"2024-04-19T02:04:09.070623+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:10.590593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.025026817,"stat_sum":{"num_bytes":7733248,"num_objects":7,"num_object_clones":4,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":27,"num_read_kb":19020,"num_write":33,"num_write_kb":10256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,10,2,9,1,7,11],"acting":[4,3,10,2,9,1,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"},{"start":"16","length":"1"},{"start":"1d","length":"1"}]},{"pgid":"3.8","version":"82'100","reported_seq":394,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-19T02:04:09.070296+0000","last_change":"2024-04-19T02:04:06.198746+0000","last_active":"2024-04-19T02:04:09.070296+0000","last_peered":"2024-04-19T02:04:09.070296+0000","last_clean":"2024-04-19T02:04:09.070296+0000","last_became_active":"2024-04-19T02:03:11.679384+0000","last_became_peered":"2024-04-19T02:03:11.679384+0000","last_unstale":"2024-04-19T02:04:09.070296+0000","last_undegraded":"2024-04-19T02:04:09.070296+0000","last_fullsized":"2024-04-19T02:04:09.070296+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:22.577801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.03098387,"stat_sum":{"num_bytes":13205504,"num_objects":8,"num_object_clones":5,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":35729,"num_write":42,"num_write_kb":16128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7,11,5,9,6,10],"acting":[4,1,7,11,5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"},{"start":"16","length":"1"},{"start":"1d","length":"1"}]},{"pgid":"1.0","version":"21'137","reported_seq":315,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-19T02:04:09.069898+0000","last_change":"2024-04-19T02:02:59.974166+0000","last_active":"2024-04-19T02:04:09.069898+0000","last_peered":"2024-04-19T02:04:09.069898+0000","last_clean":"2024-04-19T02:04:09.069898+0000","last_became_active":"2024-04-19T02:02:59.972005+0000","last_became_peered":"2024-04-19T02:02:59.972005+0000","last_unstale":"2024-04-19T02:04:09.069898+0000","last_undegraded":"2024-04-19T02:04:09.069898+0000","last_fullsized":"2024-04-19T02:04:09.069898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:02:58.486225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:02:58.486225+0000","last_clean_scrub_stamp":"2024-04-19T02:02:58.486225+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:20.275707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":133,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-19T02:04:08.061000+0000","last_change":"2024-04-19T02:03:03.026806+0000","last_active":"2024-04-19T02:04:08.061000+0000","last_peered":"2024-04-19T02:04:08.061000+0000","last_clean":"2024-04-19T02:04:08.061000+0000","last_became_active":"2024-04-19T02:03:03.026063+0000","last_became_peered":"2024-04-19T02:03:03.026063+0000","last_unstale":"2024-04-19T02:04:08.061000+0000","last_undegraded":"2024-04-19T02:04:08.061000+0000","last_fullsized":"2024-04-19T02:04:08.061000+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:19.673016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"82'200","reported_seq":489,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-19T02:04:08.092029+0000","last_change":"2024-04-19T02:04:06.207744+0000","last_active":"2024-04-19T02:04:08.092029+0000","last_peered":"2024-04-19T02:04:08.092029+0000","last_clean":"2024-04-19T02:04:08.092029+0000","last_became_active":"2024-04-19T02:03:11.630475+0000","last_became_peered":"2024-04-19T02:03:11.630475+0000","last_unstale":"2024-04-19T02:04:08.092029+0000","last_undegraded":"2024-04-19T02:04:08.092029+0000","last_fullsized":"2024-04-19T02:04:08.092029+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:13.791760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.045214945999999999,"stat_sum":{"num_bytes":13549568,"num_objects":13,"num_object_clones":7,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":60,"num_read_kb":23775,"num_write":134,"num_write_kb":23776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10,4,8,1,11,9],"acting":[3,5,10,4,8,1,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"},{"start":"16","length":"1"},{"start":"1d","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":136,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-19T02:04:09.069882+0000","last_change":"2024-04-19T02:03:02.561076+0000","last_active":"2024-04-19T02:04:09.069882+0000","last_peered":"2024-04-19T02:04:09.069882+0000","last_clean":"2024-04-19T02:04:09.069882+0000","last_became_active":"2024-04-19T02:03:02.560210+0000","last_became_peered":"2024-04-19T02:03:02.560210+0000","last_unstale":"2024-04-19T02:04:09.069882+0000","last_undegraded":"2024-04-19T02:04:09.069882+0000","last_fullsized":"2024-04-19T02:04:09.069882+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:16.900004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"82'49","reported_seq":345,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-19T02:04:10.082840+0000","last_change":"2024-04-19T02:04:10.083199+0000","last_active":"2024-04-19T02:04:10.082840+0000","last_peered":"2024-04-19T02:04:10.082840+0000","last_clean":"2024-04-19T02:04:10.082840+0000","last_became_active":"2024-04-19T02:03:11.633394+0000","last_became_peered":"2024-04-19T02:03:11.633394+0000","last_unstale":"2024-04-19T02:04:10.082840+0000","last_undegraded":"2024-04-19T02:04:10.082840+0000","last_fullsized":"2024-04-19T02:04:10.082840+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:18.943106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061024699999999996,"stat_sum":{"num_bytes":6537216,"num_objects":5,"num_object_clones":3,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":23,"num_read_kb":4042,"num_write":43,"num_write_kb":6912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,4,5,10,7,9,1],"acting":[0,11,4,5,10,7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"},{"start":"16","length":"1"},{"start":"1d","length":"1"},{"start":"21","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":129,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-19T02:04:08.091794+0000","last_change":"2024-04-19T02:03:09.594947+0000","last_active":"2024-04-19T02:04:08.091794+0000","last_peered":"2024-04-19T02:04:08.091794+0000","last_clean":"2024-04-19T02:04:08.091794+0000","last_became_active":"2024-04-19T02:03:09.594072+0000","last_became_peered":"2024-04-19T02:03:09.594072+0000","last_unstale":"2024-04-19T02:04:08.091794+0000","last_undegraded":"2024-04-19T02:04:08.091794+0000","last_fullsized":"2024-04-19T02:04:08.091794+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:29.461650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":278,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-19T02:04:08.061714+0000","last_change":"2024-04-19T02:04:06.167105+0000","last_active":"2024-04-19T02:04:08.061714+0000","last_peered":"2024-04-19T02:04:08.061714+0000","last_clean":"2024-04-19T02:04:08.061714+0000","last_became_active":"2024-04-19T02:03:11.678078+0000","last_became_peered":"2024-04-19T02:03:11.678078+0000","last_unstale":"2024-04-19T02:04:08.061714+0000","last_undegraded":"2024-04-19T02:04:08.061714+0000","last_fullsized":"2024-04-19T02:04:08.061714+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:18.366598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001007324,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,9,1,6,4,0,10],"acting":[5,7,9,1,6,4,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"},{"start":"16","length":"1"},{"start":"1d","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":136,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-19T02:04:09.070280+0000","last_change":"2024-04-19T02:03:02.560133+0000","last_active":"2024-04-19T02:04:09.070280+0000","last_peered":"2024-04-19T02:04:09.070280+0000","last_clean":"2024-04-19T02:04:09.070280+0000","last_became_active":"2024-04-19T02:03:02.559377+0000","last_became_peered":"2024-04-19T02:03:02.559377+0000","last_unstale":"2024-04-19T02:04:09.070280+0000","last_undegraded":"2024-04-19T02:04:09.070280+0000","last_fullsized":"2024-04-19T02:04:09.070280+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:30.728400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"82'39","reported_seq":319,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-19T02:04:08.084863+0000","last_change":"2024-04-19T02:04:06.212396+0000","last_active":"2024-04-19T02:04:08.084863+0000","last_peered":"2024-04-19T02:04:08.084863+0000","last_clean":"2024-04-19T02:04:08.084863+0000","last_became_active":"2024-04-19T02:03:11.677099+0000","last_became_peered":"2024-04-19T02:03:11.677099+0000","last_unstale":"2024-04-19T02:04:08.084863+0000","last_undegraded":"2024-04-19T02:04:08.084863+0000","last_fullsized":"2024-04-19T02:04:08.084863+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:23.957818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032936960000000001,"stat_sum":{"num_bytes":4390912,"num_objects":5,"num_object_clones":3,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":11,"num_read_kb":2370,"num_write":57,"num_write_kb":4288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,0,3,4,7,9,6],"acting":[10,8,0,3,4,7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"},{"start":"16","length":"1"},{"start":"1d","length":"1"}]},{"pgid":"3.5","version":"80'65","reported_seq":342,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-19T02:04:08.084508+0000","last_change":"2024-04-19T02:04:06.183366+0000","last_active":"2024-04-19T02:04:08.084508+0000","last_peered":"2024-04-19T02:04:08.084508+0000","last_clean":"2024-04-19T02:04:08.084508+0000","last_became_active":"2024-04-19T02:03:11.728949+0000","last_became_peered":"2024-04-19T02:03:11.728949+0000","last_unstale":"2024-04-19T02:04:08.084508+0000","last_undegraded":"2024-04-19T02:04:08.084508+0000","last_fullsized":"2024-04-19T02:04:08.084508+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:15.024914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042794080000000002,"stat_sum":{"num_bytes":4964352,"num_objects":6,"num_object_clones":3,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":8,"num_write":48,"num_write_kb":6192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2,9,8,1,6,5],"acting":[10,3,2,9,8,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"},{"start":"16","length":"1"},{"start":"1d","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":134,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-19T02:04:08.057941+0000","last_change":"2024-04-19T02:03:02.597582+0000","last_active":"2024-04-19T02:04:08.057941+0000","last_peered":"2024-04-19T02:04:08.057941+0000","last_clean":"2024-04-19T02:04:08.057941+0000","last_became_active":"2024-04-19T02:03:02.597118+0000","last_became_peered":"2024-04-19T02:03:02.597118+0000","last_unstale":"2024-04-19T02:04:08.057941+0000","last_undegraded":"2024-04-19T02:04:08.057941+0000","last_fullsized":"2024-04-19T02:04:08.057941+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:13.683143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"84'135","reported_seq":409,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-19T02:04:08.088271+0000","last_change":"2024-04-19T02:04:06.205433+0000","last_active":"2024-04-19T02:04:08.088271+0000","last_peered":"2024-04-19T02:04:08.088271+0000","last_clean":"2024-04-19T02:04:08.088271+0000","last_became_active":"2024-04-19T02:03:11.730165+0000","last_became_peered":"2024-04-19T02:03:11.730165+0000","last_unstale":"2024-04-19T02:04:08.088271+0000","last_undegraded":"2024-04-19T02:04:08.088271+0000","last_fullsized":"2024-04-19T02:04:08.088271+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:23.196103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033014798999999997,"stat_sum":{"num_bytes":8323072,"num_objects":15,"num_object_clones":10,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":31,"num_read_kb":6448,"num_write":92,"num_write_kb":12304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,10,8,7,2,11,0],"acting":[1,5,10,8,7,2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"},{"start":"16","length":"1"},{"start":"1d","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":142,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-19T02:04:08.086433+0000","last_change":"2024-04-19T02:04:04.753289+0000","last_active":"2024-04-19T02:04:08.086433+0000","last_peered":"2024-04-19T02:04:08.086433+0000","last_clean":"2024-04-19T02:04:08.086433+0000","last_became_active":"2024-04-19T02:03:02.595232+0000","last_became_peered":"2024-04-19T02:03:02.595232+0000","last_unstale":"2024-04-19T02:04:08.086433+0000","last_undegraded":"2024-04-19T02:04:08.086433+0000","last_fullsized":"2024-04-19T02:04:08.086433+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:04.752922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:04.752922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:25.635349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"80'30","reported_seq":307,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-19T02:04:08.092288+0000","last_change":"2024-04-19T02:04:06.185530+0000","last_active":"2024-04-19T02:04:08.092288+0000","last_peered":"2024-04-19T02:04:08.092288+0000","last_clean":"2024-04-19T02:04:08.092288+0000","last_became_active":"2024-04-19T02:03:11.675056+0000","last_became_peered":"2024-04-19T02:03:11.675056+0000","last_unstale":"2024-04-19T02:04:08.092288+0000","last_undegraded":"2024-04-19T02:04:08.092288+0000","last_fullsized":"2024-04-19T02:04:08.092288+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:11.998553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022571230000000001,"stat_sum":{"num_bytes":2162688,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":5,"num_read_kb":2,"num_write":9,"num_write_kb":2496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,9,0,7,5,11,1],"acting":[3,6,9,0,7,5,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"},{"start":"16","length":"1"},{"start":"1d","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":134,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-19T02:04:08.057788+0000","last_change":"2024-04-19T02:03:02.988302+0000","last_active":"2024-04-19T02:04:08.057788+0000","last_peered":"2024-04-19T02:04:08.057788+0000","last_clean":"2024-04-19T02:04:08.057788+0000","last_became_active":"2024-04-19T02:03:02.988073+0000","last_became_peered":"2024-04-19T02:03:02.988073+0000","last_unstale":"2024-04-19T02:04:08.057788+0000","last_undegraded":"2024-04-19T02:04:08.057788+0000","last_fullsized":"2024-04-19T02:04:08.057788+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:28.455271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"82'13","reported_seq":302,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-19T02:04:10.084513+0000","last_change":"2024-04-19T02:04:10.085003+0000","last_active":"2024-04-19T02:04:10.084513+0000","last_peered":"2024-04-19T02:04:10.084513+0000","last_clean":"2024-04-19T02:04:10.084513+0000","last_became_active":"2024-04-19T02:03:11.673940+0000","last_became_peered":"2024-04-19T02:03:11.673940+0000","last_unstale":"2024-04-19T02:04:10.084513+0000","last_undegraded":"2024-04-19T02:04:10.084513+0000","last_fullsized":"2024-04-19T02:04:10.084513+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:10.585869+0000","last_clean_scrub_stamp":"2024-04-19T02:03:10.585869+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:36.950341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071609099999999997,"stat_sum":{"num_bytes":3063808,"num_objects":3,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":530,"num_write":24,"num_write_kb":2992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9,8,4,5,6,10],"acting":[0,1,9,8,4,5,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"12","length":"1"},{"start":"16","length":"1"},{"start":"1d","length":"1"},{"start":"21","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":136,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-19T02:04:09.070522+0000","last_change":"2024-04-19T02:03:02.570839+0000","last_active":"2024-04-19T02:04:09.070522+0000","last_peered":"2024-04-19T02:04:09.070522+0000","last_clean":"2024-04-19T02:04:09.070522+0000","last_became_active":"2024-04-19T02:03:02.570116+0000","last_became_peered":"2024-04-19T02:03:02.570116+0000","last_unstale":"2024-04-19T02:04:09.070522+0000","last_undegraded":"2024-04-19T02:04:09.070522+0000","last_fullsized":"2024-04-19T02:04:09.070522+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:03:01.531230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:04:23.134425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":126418944,"num_objects":108,"num_object_clones":61,"num_object_copies":864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":108,"num_whiteouts":7,"num_read":441,"num_read_kb":238457,"num_write":786,"num_write_kb":159504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":252936192,"data_stored":431095808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1197,"ondisk_log_size":1197,"up":128,"acting":128,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":19,"seq":81604378642,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47500,"kb_used_data":16316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324340,"statfs":{"total":96636764160,"available":96588124160,"internally_reserved":0,"allocated":16707584,"data_stored":27343294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":19,"seq":81604378641,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49064,"kb_used_data":17880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322776,"statfs":{"total":96636764160,"available":96586522624,"internally_reserved":0,"allocated":18309120,"data_stored":31075063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411345,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76016,"kb_used_data":26784,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94295824,"statfs":{"total":96636764160,"available":96558923776,"internally_reserved":0,"allocated":27426816,"data_stored":45689037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411346,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74984,"kb_used_data":25752,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94296856,"statfs":{"total":96636764160,"available":96559980544,"internally_reserved":0,"allocated":26370048,"data_stored":43067597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411346,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70472,"kb_used_data":21240,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94301368,"statfs":{"total":96636764160,"available":96564600832,"internally_reserved":0,"allocated":21749760,"data_stored":37424094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411346,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57724,"kb_used_data":26544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94314116,"statfs":{"total":96636764160,"available":96577654784,"internally_reserved":0,"allocated":27181056,"data_stored":45328589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":18,"seq":77309411344,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36748,"kb_used_data":9728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335092,"statfs":{"total":96636764160,"available":96599134208,"internally_reserved":0,"allocated":9961472,"data_stored":15227085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411345,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54896,"kb_used_data":23716,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94316944,"statfs":{"total":96636764160,"available":96580550656,"internally_reserved":0,"allocated":24285184,"data_stored":38451405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":18,"seq":77309411345,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73440,"kb_used_data":24212,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94298400,"statfs":{"total":96636764160,"available":96561561600,"internally_reserved":0,"allocated":24793088,"data_stored":39233741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":17,"seq":73014444049,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49052,"kb_used_data":17864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322788,"statfs":{"total":96636764160,"available":96586534912,"internally_reserved":0,"allocated":18292736,"data_stored":30595036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":17,"seq":73014444049,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76612,"kb_used_data":27376,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94295228,"statfs":{"total":96636764160,"available":96558313472,"internally_reserved":0,"allocated":28033024,"data_stored":47426285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":17,"seq":73014444049,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70872,"kb_used_data":21644,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94300968,"statfs":{"total":96636764160,"available":96564191232,"internally_reserved":0,"allocated":22163456,"data_stored":35735516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17313792,"data_stored":30457856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21356544,"data_stored":35909632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":15822848,"data_stored":27979776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":23822336,"data_stored":38633472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":23314432,"data_stored":37851136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":26529792,"data_stored":46297088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8990720,"data_stored":14626816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":26210304,"data_stored":44728320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20234240,"data_stored":36286464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":25399296,"data_stored":42467328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":17485824,"data_stored":30769152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":26456064,"data_stored":45088768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T02:04:12.815 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-04-19T02:04:12.815 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-04-19T02:04:12.856 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:04:12.857+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:04:12.866 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:12.867 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:12.880 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-19T02:04:12.951 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-04-19T02:04:12.949+0000 7ff7b1535640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 24952) UID: 0 2024-04-19T02:04:13.004 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:13.004 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:13.018 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-19T02:04:13.042 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-19T02:04:13.062 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:04:13.064+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:04:13.136 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:13.136 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:13.154 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-19T02:04:13.184 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0: {} 2024-04-19T02:04:13.185 INFO:teuthology.orchestra.run.smithi059.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-19T02:04:13.198 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:04:13.200+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:04:13.201 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1: {} 2024-04-19T02:04:13.202 INFO:teuthology.orchestra.run.smithi059.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-19T02:04:13.219 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2: {} 2024-04-19T02:04:13.219 INFO:teuthology.orchestra.run.smithi059.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-19T02:04:13.237 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3: {} 2024-04-19T02:04:13.237 INFO:teuthology.orchestra.run.smithi059.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-19T02:04:13.256 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-04-19T02:04:13.257+0000 7f07d8953640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 24950) UID: 0 2024-04-19T02:04:13.257 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4: {} 2024-04-19T02:04:13.257 INFO:teuthology.orchestra.run.smithi059.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-19T02:04:13.278 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5: {} 2024-04-19T02:04:13.278 INFO:teuthology.orchestra.run.smithi059.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-19T02:04:13.298 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6: {} 2024-04-19T02:04:13.299 INFO:teuthology.orchestra.run.smithi059.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-19T02:04:13.319 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7: {} 2024-04-19T02:04:13.319 INFO:teuthology.orchestra.run.smithi059.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-19T02:04:13.320 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-04-19T02:04:13.321 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-19T02:04:13.337 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-19T02:04:13.340 INFO:teuthology.orchestra.run.smithi059.stdout:osd.8: {} 2024-04-19T02:04:13.340 INFO:teuthology.orchestra.run.smithi059.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-19T02:04:13.360 INFO:teuthology.orchestra.run.smithi059.stdout:osd.9: {} 2024-04-19T02:04:13.360 INFO:teuthology.orchestra.run.smithi059.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-19T02:04:13.378 INFO:teuthology.orchestra.run.smithi059.stdout:osd.10: {} 2024-04-19T02:04:13.378 INFO:teuthology.orchestra.run.smithi059.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-19T02:04:13.382 INFO:tasks.ceph.osd.8.smithi179.stderr:2024-04-19T02:04:13.384+0000 7f2b67ccf640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25305) UID: 0 2024-04-19T02:04:13.396 INFO:teuthology.orchestra.run.smithi059.stdout:osd.11: {} 2024-04-19T02:04:13.397 INFO:teuthology.orchestra.run.smithi059.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-19T02:04:13.453 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:04:13.453 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:04:13.453 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-04-19T02:04:13.453 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:04:13.459 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-04-19T02:04:13.457+0000 7f9b39675640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 24945) UID: 0 2024-04-19T02:04:13.468 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-19T02:04:13.559 INFO:tasks.ceph.osd.9.smithi179.stderr:2024-04-19T02:04:13.560+0000 7ff23f045640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 25306) UID: 0 2024-04-19T02:04:13.634 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:04:13.634 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:04:13.634 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-04-19T02:04:13.635 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-04-19T02:04:13.635 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:04:13.648 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-19T02:04:13.690 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:04:13.692+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:04:13.760 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-04-19T02:04:13.761+0000 7f254af2d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24948) UID: 0 2024-04-19T02:04:13.815 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:04:13.815 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-04-19T02:04:13.815 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-04-19T02:04:13.816 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-04-19T02:04:13.816 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.816 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.1s4 47/26 ECSubWrite(tid=29, reqid=osd.0.0:29, at_version=47'15, trim_to=0'0, roll_forward_to=46'13))", 2024-04-19T02:04:13.816 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.087131+0000", 2024-04-19T02:04:13.816 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 42.727961776999997, 2024-04-19T02:04:13.816 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.028478961000000001, 2024-04-19T02:04:13.816 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:13.816 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:13.816 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:04:13.816 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:13.816 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.816 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:13.816 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087131+0000", 2024-04-19T02:04:13.816 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.817 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.817 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.817 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:13.817 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087131+0000", 2024-04-19T02:04:13.817 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.817 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.817 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.817 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:13.817 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087132+0000", 2024-04-19T02:04:13.817 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4000000000000001e-07 2024-04-19T02:04:13.817 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.817 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.817 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:13.817 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087185+0000", 2024-04-19T02:04:13.817 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3016999999999997e-05 2024-04-19T02:04:13.818 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.818 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.818 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:13.818 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087232+0000", 2024-04-19T02:04:13.818 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6989e-05 2024-04-19T02:04:13.818 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.818 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.818 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:13.818 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087259+0000", 2024-04-19T02:04:13.818 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7199000000000002e-05 2024-04-19T02:04:13.818 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.818 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.818 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:13.818 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087431+0000", 2024-04-19T02:04:13.818 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017193199999999999 2024-04-19T02:04:13.819 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.819 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.819 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:13.819 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087457+0000", 2024-04-19T02:04:13.819 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6148000000000001e-05 2024-04-19T02:04:13.819 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.819 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.819 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:13.819 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.115506+0000", 2024-04-19T02:04:13.819 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.028048812999999999 2024-04-19T02:04:13.819 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.819 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.819 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:13.819 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.115610+0000", 2024-04-19T02:04:13.819 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010412300000000001 2024-04-19T02:04:13.820 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.820 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:13.820 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.820 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.820 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.820 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.1s4 47/26 ECSubWrite(tid=30, reqid=osd.0.0:30, at_version=47'17, trim_to=0'0, roll_forward_to=46'13))", 2024-04-19T02:04:13.820 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.087466+0000", 2024-04-19T02:04:13.820 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 42.727627390000002, 2024-04-19T02:04:13.820 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.030894781, 2024-04-19T02:04:13.820 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:13.820 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:13.820 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:04:13.820 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:13.820 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.820 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:13.821 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087466+0000", 2024-04-19T02:04:13.821 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.821 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.821 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.821 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:13.821 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087466+0000", 2024-04-19T02:04:13.821 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.821 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.821 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.821 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:13.821 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087466+0000", 2024-04-19T02:04:13.821 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0400000000000005e-07 2024-04-19T02:04:13.821 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.821 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.821 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:13.822 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087524+0000", 2024-04-19T02:04:13.822 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7632e-05 2024-04-19T02:04:13.822 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.822 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.822 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:13.822 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087550+0000", 2024-04-19T02:04:13.822 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5871999999999999e-05 2024-04-19T02:04:13.822 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.822 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.822 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:13.822 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.087570+0000", 2024-04-19T02:04:13.822 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0531999999999999e-05 2024-04-19T02:04:13.822 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.822 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.823 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:13.823 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.091486+0000", 2024-04-19T02:04:13.823 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039151009999999998 2024-04-19T02:04:13.823 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.823 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.823 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:13.823 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.091519+0000", 2024-04-19T02:04:13.823 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3874999999999997e-05 2024-04-19T02:04:13.823 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.823 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.823 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:13.823 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.118273+0000", 2024-04-19T02:04:13.823 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.026753835 2024-04-19T02:04:13.823 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.824 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.824 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:13.824 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.118360+0000", 2024-04-19T02:04:13.824 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7029999999999999e-05 2024-04-19T02:04:13.824 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.824 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:13.824 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.824 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.824 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.824 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s2 47/26 ECSubWrite(tid=181, reqid=osd.3.0:181, at_version=47'87, trim_to=0'0, roll_forward_to=46'85))", 2024-04-19T02:04:13.824 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.088687+0000", 2024-04-19T02:04:13.824 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 42.726406005999998, 2024-04-19T02:04:13.824 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.027025486000000001, 2024-04-19T02:04:13.824 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:13.824 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:13.824 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:04:13.825 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:13.825 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.825 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:13.825 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088687+0000", 2024-04-19T02:04:13.825 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.825 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.825 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.825 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:13.825 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088687+0000", 2024-04-19T02:04:13.825 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.825 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.825 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.825 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:13.825 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088687+0000", 2024-04-19T02:04:13.825 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7999999999999996e-07 2024-04-19T02:04:13.826 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.826 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.826 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:13.826 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088786+0000", 2024-04-19T02:04:13.826 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8510000000000004e-05 2024-04-19T02:04:13.826 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.826 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.826 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:13.826 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088822+0000", 2024-04-19T02:04:13.826 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6284e-05 2024-04-19T02:04:13.826 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.826 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.826 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:13.826 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088846+0000", 2024-04-19T02:04:13.826 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4125999999999999e-05 2024-04-19T02:04:13.827 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.827 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.827 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:13.827 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089107+0000", 2024-04-19T02:04:13.827 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00026044799999999998 2024-04-19T02:04:13.827 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.827 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.827 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:13.827 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089134+0000", 2024-04-19T02:04:13.827 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7067999999999999e-05 2024-04-19T02:04:13.827 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.827 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.827 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:13.827 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.115620+0000", 2024-04-19T02:04:13.827 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.026486464000000001 2024-04-19T02:04:13.828 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.828 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.828 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:13.828 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.115712+0000", 2024-04-19T02:04:13.828 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2106000000000002e-05 2024-04-19T02:04:13.828 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.828 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:13.828 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.828 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.828 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.828 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.bs5 47/26 ECSubWrite(tid=180, reqid=osd.3.0:180, at_version=47'36, trim_to=0'0, roll_forward_to=46'34))", 2024-04-19T02:04:13.828 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.088907+0000", 2024-04-19T02:04:13.828 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 42.726186396999999, 2024-04-19T02:04:13.828 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.028441793999999999, 2024-04-19T02:04:13.828 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:13.829 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:13.829 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:04:13.829 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:13.829 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.829 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:13.829 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088907+0000", 2024-04-19T02:04:13.829 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.829 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.829 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.829 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:13.829 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088907+0000", 2024-04-19T02:04:13.829 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.829 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.829 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.829 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:13.830 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088908+0000", 2024-04-19T02:04:13.830 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0100000000000001e-06 2024-04-19T02:04:13.830 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.830 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.830 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:13.830 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088952+0000", 2024-04-19T02:04:13.830 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4613000000000001e-05 2024-04-19T02:04:13.830 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.830 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.830 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:13.830 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088975+0000", 2024-04-19T02:04:13.830 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2518999999999999e-05 2024-04-19T02:04:13.830 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.830 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.831 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:13.831 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.088993+0000", 2024-04-19T02:04:13.831 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7855000000000001e-05 2024-04-19T02:04:13.831 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.831 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.831 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:13.831 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089130+0000", 2024-04-19T02:04:13.831 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013709099999999999 2024-04-19T02:04:13.831 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.831 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.831 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:13.831 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089155+0000", 2024-04-19T02:04:13.831 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4924999999999999e-05 2024-04-19T02:04:13.831 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.831 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.831 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:13.832 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.117263+0000", 2024-04-19T02:04:13.832 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.028107953000000001 2024-04-19T02:04:13.832 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.832 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.832 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:13.832 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.117348+0000", 2024-04-19T02:04:13.832 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5828000000000006e-05 2024-04-19T02:04:13.832 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.832 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:13.832 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.832 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.832 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.832 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s2 47/26 ECSubWrite(tid=182, reqid=osd.3.0:182, at_version=47'89, trim_to=0'0, roll_forward_to=46'85))", 2024-04-19T02:04:13.833 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.089036+0000", 2024-04-19T02:04:13.833 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 42.726057369000003, 2024-04-19T02:04:13.833 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.027692491, 2024-04-19T02:04:13.833 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:13.833 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:13.833 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:04:13.833 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:13.833 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.833 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:13.833 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089036+0000", 2024-04-19T02:04:13.833 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.833 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.833 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.833 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:13.833 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089036+0000", 2024-04-19T02:04:13.834 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.834 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.834 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.834 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:13.834 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089036+0000", 2024-04-19T02:04:13.834 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7599999999999996e-07 2024-04-19T02:04:13.834 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.834 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.834 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:13.834 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089097+0000", 2024-04-19T02:04:13.834 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0223000000000003e-05 2024-04-19T02:04:13.834 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.834 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.834 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:13.834 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089117+0000", 2024-04-19T02:04:13.835 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0724999999999999e-05 2024-04-19T02:04:13.835 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.835 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.835 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:13.835 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089134+0000", 2024-04-19T02:04:13.835 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7065e-05 2024-04-19T02:04:13.835 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.835 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.835 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:13.835 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.091684+0000", 2024-04-19T02:04:13.835 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0025499780000000001 2024-04-19T02:04:13.835 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.835 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.835 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:13.836 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.091723+0000", 2024-04-19T02:04:13.836 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8787e-05 2024-04-19T02:04:13.836 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.836 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.836 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:13.836 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.116611+0000", 2024-04-19T02:04:13.836 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.024888321000000001 2024-04-19T02:04:13.836 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.836 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.836 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:13.836 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.116728+0000", 2024-04-19T02:04:13.836 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000116616 2024-04-19T02:04:13.836 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.836 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:13.836 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.837 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.837 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.837 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.bs5 47/26 ECSubWrite(tid=183, reqid=osd.3.0:183, at_version=47'38, trim_to=0'0, roll_forward_to=46'34))", 2024-04-19T02:04:13.837 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.089162+0000", 2024-04-19T02:04:13.837 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 42.725931002000003, 2024-04-19T02:04:13.837 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.031704567000000003, 2024-04-19T02:04:13.837 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:13.837 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:13.837 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:04:13.837 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:13.837 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.837 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:13.837 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089162+0000", 2024-04-19T02:04:13.837 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.837 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.838 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.838 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:13.838 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089162+0000", 2024-04-19T02:04:13.838 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.838 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.838 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.838 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:13.838 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089163+0000", 2024-04-19T02:04:13.838 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8999999999999996e-07 2024-04-19T02:04:13.838 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.838 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.838 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:13.838 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089208+0000", 2024-04-19T02:04:13.838 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5670999999999997e-05 2024-04-19T02:04:13.839 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.839 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.839 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:13.839 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089231+0000", 2024-04-19T02:04:13.839 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2898999999999999e-05 2024-04-19T02:04:13.839 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.839 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.839 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:13.839 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.089248+0000", 2024-04-19T02:04:13.839 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7170999999999999e-05 2024-04-19T02:04:13.839 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.839 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.839 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:13.839 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.091699+0000", 2024-04-19T02:04:13.839 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0024503010000000002 2024-04-19T02:04:13.840 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.840 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.840 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:13.840 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.091731+0000", 2024-04-19T02:04:13.840 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2301000000000001e-05 2024-04-19T02:04:13.840 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.840 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.840 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:13.840 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.120784+0000", 2024-04-19T02:04:13.840 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.029052923000000001 2024-04-19T02:04:13.840 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.840 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.840 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:13.840 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.120866+0000", 2024-04-19T02:04:13.840 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2610999999999999e-05 2024-04-19T02:04:13.841 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.841 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:13.841 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.841 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.841 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.841 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.8s7 47/26 ECSubWrite(tid=86, reqid=osd.4.0:86, at_version=47'44, trim_to=0'0, roll_forward_to=46'42))", 2024-04-19T02:04:13.841 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:31.092233+0000", 2024-04-19T02:04:13.841 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 42.722860386000001, 2024-04-19T02:04:13.841 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.026209364999999998, 2024-04-19T02:04:13.841 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:13.841 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:13.841 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:04:13.841 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:13.841 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.841 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:13.842 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092233+0000", 2024-04-19T02:04:13.842 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.842 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.842 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.842 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:13.842 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092233+0000", 2024-04-19T02:04:13.842 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.842 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.842 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.842 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:13.842 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092233+0000", 2024-04-19T02:04:13.842 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8699999999999995e-07 2024-04-19T02:04:13.842 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.842 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.843 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:13.843 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092282+0000", 2024-04-19T02:04:13.843 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8869999999999998e-05 2024-04-19T02:04:13.843 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.843 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.843 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:13.843 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092311+0000", 2024-04-19T02:04:13.843 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9201e-05 2024-04-19T02:04:13.843 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.843 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.843 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:13.843 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092342+0000", 2024-04-19T02:04:13.843 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1275000000000001e-05 2024-04-19T02:04:13.843 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.844 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.844 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:13.844 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092597+0000", 2024-04-19T02:04:13.844 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00025494899999999997 2024-04-19T02:04:13.844 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.844 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.844 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:13.844 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.092622+0000", 2024-04-19T02:04:13.844 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4355e-05 2024-04-19T02:04:13.844 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.844 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.844 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:13.844 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.118397+0000", 2024-04-19T02:04:13.844 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.025775005 2024-04-19T02:04:13.845 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.845 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.845 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:13.845 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:31.118442+0000", 2024-04-19T02:04:13.845 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5123000000000001e-05 2024-04-19T02:04:13.845 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.845 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:13.845 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.845 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.845 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.845 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:586 3.3s0 3:c26f510a:::smithi05930959-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56)", 2024-04-19T02:04:13.845 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:40.218545+0000", 2024-04-19T02:04:13.845 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 33.596548341000002, 2024-04-19T02:04:13.845 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.023206135999999999, 2024-04-19T02:04:13.845 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:13.846 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:13.846 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:04:13.846 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:13.846 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:13.846 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:13.846 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 586 2024-04-19T02:04:13.846 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.846 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:13.846 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.846 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:13.846 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218545+0000", 2024-04-19T02:04:13.846 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.846 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.846 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.846 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:13.847 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218545+0000", 2024-04-19T02:04:13.847 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.847 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.847 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.847 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:13.847 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218546+0000", 2024-04-19T02:04:13.847 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4160000000000001e-06 2024-04-19T02:04:13.847 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.847 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.847 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:13.847 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218553+0000", 2024-04-19T02:04:13.847 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1770000000000001e-06 2024-04-19T02:04:13.847 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.847 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.847 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:13.847 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218577+0000", 2024-04-19T02:04:13.848 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3635e-05 2024-04-19T02:04:13.848 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.848 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.850 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:13.850 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.218593+0000", 2024-04-19T02:04:13.850 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5892999999999998e-05 2024-04-19T02:04:13.850 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.850 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.850 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:13.850 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.234991+0000", 2024-04-19T02:04:13.850 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016397954999999999 2024-04-19T02:04:13.850 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.850 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.850 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:04:13.850 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.235250+0000", 2024-04-19T02:04:13.850 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000259547 2024-04-19T02:04:13.850 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.851 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.851 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:13.851 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.236766+0000", 2024-04-19T02:04:13.851 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0015157599999999999 2024-04-19T02:04:13.851 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.851 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.851 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:13.851 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.237008+0000", 2024-04-19T02:04:13.851 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00024161300000000001 2024-04-19T02:04:13.851 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.851 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.851 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:13.851 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:40.241751+0000", 2024-04-19T02:04:13.851 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0047431399999999999 2024-04-19T02:04:13.851 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.852 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:13.852 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.852 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.852 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.852 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:765 3.5s0 3:a4ae10cb:::smithi05930959-29:head [append 0~770048] snapc 17=[17,16,15,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-19T02:04:13.852 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:51.089515+0000", 2024-04-19T02:04:13.852 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 22.725578333000001, 2024-04-19T02:04:13.852 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.039942798000000002, 2024-04-19T02:04:13.852 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:13.852 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:13.852 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:13.852 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:13.852 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:13.852 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:13.853 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 765 2024-04-19T02:04:13.853 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.853 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:13.853 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.853 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:13.853 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.089515+0000", 2024-04-19T02:04:13.853 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.853 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.853 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.853 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:13.853 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.089515+0000", 2024-04-19T02:04:13.853 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.853 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.853 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.853 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:13.854 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.089518+0000", 2024-04-19T02:04:13.854 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.534e-06 2024-04-19T02:04:13.854 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.854 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.854 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:13.854 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.090847+0000", 2024-04-19T02:04:13.854 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0013285630000000001 2024-04-19T02:04:13.854 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.854 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.854 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:13.854 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.090883+0000", 2024-04-19T02:04:13.854 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6313000000000003e-05 2024-04-19T02:04:13.854 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.855 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.855 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:13.855 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.090908+0000", 2024-04-19T02:04:13.855 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5471999999999999e-05 2024-04-19T02:04:13.855 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.855 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.855 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:13.855 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.091083+0000", 2024-04-19T02:04:13.855 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000174403 2024-04-19T02:04:13.855 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.855 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.855 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op must wait for map", 2024-04-19T02:04:13.855 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.091096+0000", 2024-04-19T02:04:13.855 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3032e-05 2024-04-19T02:04:13.855 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.855 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.856 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:13.856 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.094803+0000", 2024-04-19T02:04:13.856 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0037069170000000001 2024-04-19T02:04:13.856 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.856 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.856 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:13.856 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.094961+0000", 2024-04-19T02:04:13.856 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015769699999999999 2024-04-19T02:04:13.856 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.856 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.856 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:13.856 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.097867+0000", 2024-04-19T02:04:13.856 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0029062480000000002 2024-04-19T02:04:13.856 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.856 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.857 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:13.857 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.116766+0000", 2024-04-19T02:04:13.857 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018899112999999999 2024-04-19T02:04:13.857 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.857 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.857 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:04:13.857 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.129316+0000", 2024-04-19T02:04:13.857 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012550318 2024-04-19T02:04:13.857 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.857 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.857 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:13.857 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.129457+0000", 2024-04-19T02:04:13.857 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014118799999999999 2024-04-19T02:04:13.858 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.858 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:13.858 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.858 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.858 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.858 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:766 3.5s0 3:a4ae10cb:::smithi05930959-29:head [append 0~540672] snapc 17=[17,16,15,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-19T02:04:13.858 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:51.093545+0000", 2024-04-19T02:04:13.858 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 22.721547549, 2024-04-19T02:04:13.858 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.040408039, 2024-04-19T02:04:13.858 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:13.858 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:13.858 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:13.858 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:13.858 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:13.859 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:13.859 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 766 2024-04-19T02:04:13.859 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.859 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:13.859 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.859 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:13.859 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.093545+0000", 2024-04-19T02:04:13.859 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.859 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.859 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.859 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:13.859 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.093545+0000", 2024-04-19T02:04:13.859 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.859 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.859 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.860 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:13.860 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.093548+0000", 2024-04-19T02:04:13.860 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.503e-06 2024-04-19T02:04:13.860 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.860 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.860 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:13.860 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.095040+0000", 2024-04-19T02:04:13.860 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0014918609999999999 2024-04-19T02:04:13.860 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.860 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.860 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:13.860 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.095062+0000", 2024-04-19T02:04:13.860 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1916000000000002e-05 2024-04-19T02:04:13.860 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.860 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.861 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:13.861 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.095076+0000", 2024-04-19T02:04:13.861 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4664999999999999e-05 2024-04-19T02:04:13.861 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.861 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.861 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:13.861 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.101526+0000", 2024-04-19T02:04:13.861 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0064491979999999997 2024-04-19T02:04:13.861 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.861 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.861 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:13.861 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.101864+0000", 2024-04-19T02:04:13.861 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00033830999999999998 2024-04-19T02:04:13.861 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.861 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.861 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:13.861 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.104979+0000", 2024-04-19T02:04:13.862 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0031149459999999999 2024-04-19T02:04:13.862 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.862 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.862 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:13.862 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.116769+0000", 2024-04-19T02:04:13.862 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011790512 2024-04-19T02:04:13.862 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.862 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.862 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:04:13.862 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.133770+0000", 2024-04-19T02:04:13.862 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017000347999999998 2024-04-19T02:04:13.862 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.862 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.862 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:13.863 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.133953+0000", 2024-04-19T02:04:13.863 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00018378000000000001 2024-04-19T02:04:13.863 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.863 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:13.863 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.863 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.863 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.863 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:767 3.5s0 3:a4ae10cb:::smithi05930959-29:head [append 0~753664] snapc 17=[17,16,15,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-19T02:04:13.863 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:51.099323+0000", 2024-04-19T02:04:13.863 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 22.715770355, 2024-04-19T02:04:13.863 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.035037549000000001, 2024-04-19T02:04:13.863 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:13.863 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:13.863 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:13.863 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:13.863 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:13.864 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:13.864 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 767 2024-04-19T02:04:13.864 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.864 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:13.864 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.864 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:13.864 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.099323+0000", 2024-04-19T02:04:13.864 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.864 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.864 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.864 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:13.864 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.099323+0000", 2024-04-19T02:04:13.864 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.864 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.864 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.864 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:13.864 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.099333+0000", 2024-04-19T02:04:13.865 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0412e-05 2024-04-19T02:04:13.865 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.865 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.865 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:13.865 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.100798+0000", 2024-04-19T02:04:13.865 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001465166 2024-04-19T02:04:13.865 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.865 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.865 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:13.865 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.100841+0000", 2024-04-19T02:04:13.865 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2837999999999999e-05 2024-04-19T02:04:13.865 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.865 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.865 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:13.865 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.100865+0000", 2024-04-19T02:04:13.865 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4241999999999999e-05 2024-04-19T02:04:13.866 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.866 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.866 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:13.866 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.107634+0000", 2024-04-19T02:04:13.866 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0067688389999999996 2024-04-19T02:04:13.866 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.866 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.866 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:13.866 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.107945+0000", 2024-04-19T02:04:13.866 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00031116999999999999 2024-04-19T02:04:13.866 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.866 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.866 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:13.866 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.114382+0000", 2024-04-19T02:04:13.867 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0064363579999999997 2024-04-19T02:04:13.867 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.867 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.867 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:13.867 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.118042+0000", 2024-04-19T02:04:13.867 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0036606849999999999 2024-04-19T02:04:13.867 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.867 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.867 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:04:13.867 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.134197+0000", 2024-04-19T02:04:13.867 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016155124999999999 2024-04-19T02:04:13.867 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.867 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.867 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:13.867 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.134360+0000", 2024-04-19T02:04:13.867 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000162714 2024-04-19T02:04:13.868 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.868 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:13.868 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.868 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.868 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.868 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:768 3.5s0 3:a4ae10cb:::smithi05930959-29:head [append 0~262144] snapc 17=[17,16,15,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-19T02:04:13.868 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:51.101861+0000", 2024-04-19T02:04:13.868 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 22.713231845999999, 2024-04-19T02:04:13.868 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.032826545999999998, 2024-04-19T02:04:13.868 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:13.868 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:13.868 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:13.868 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:13.868 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:13.868 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:13.869 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 768 2024-04-19T02:04:13.869 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.869 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:13.869 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.869 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:13.869 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.101861+0000", 2024-04-19T02:04:13.869 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.869 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.869 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.869 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:13.869 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.101861+0000", 2024-04-19T02:04:13.869 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.869 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.869 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.869 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:13.869 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.101866+0000", 2024-04-19T02:04:13.870 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4399999999999998e-06 2024-04-19T02:04:13.870 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.870 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.870 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:13.870 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102379+0000", 2024-04-19T02:04:13.870 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00051389600000000001 2024-04-19T02:04:13.870 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.870 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.870 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:13.870 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102412+0000", 2024-04-19T02:04:13.870 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2832e-05 2024-04-19T02:04:13.870 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.870 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.870 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:13.871 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102439+0000", 2024-04-19T02:04:13.871 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6842000000000002e-05 2024-04-19T02:04:13.871 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.871 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.871 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:13.871 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.118803+0000", 2024-04-19T02:04:13.871 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016363637 2024-04-19T02:04:13.871 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.871 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.871 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:13.871 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.119080+0000", 2024-04-19T02:04:13.871 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00027685600000000002 2024-04-19T02:04:13.871 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.871 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.872 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:13.872 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.121701+0000", 2024-04-19T02:04:13.872 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0026218389999999999 2024-04-19T02:04:13.872 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.872 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.872 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:13.872 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.127449+0000", 2024-04-19T02:04:13.872 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.005747149 2024-04-19T02:04:13.872 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.872 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.872 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:04:13.872 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.134572+0000", 2024-04-19T02:04:13.872 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0071231269999999999 2024-04-19T02:04:13.872 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.872 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.873 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:13.873 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.134688+0000", 2024-04-19T02:04:13.873 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011592800000000001 2024-04-19T02:04:13.873 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.873 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:13.873 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.873 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.873 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.873 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:769 3.5s0 3:a4ae10cb:::smithi05930959-29:head [setxattr _header (57) in=7b] snapc 17=[17,16,15,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-19T02:04:13.873 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:51.102479+0000", 2024-04-19T02:04:13.873 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 22.712613894, 2024-04-19T02:04:13.873 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.032545561000000001, 2024-04-19T02:04:13.873 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:13.873 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:13.874 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:13.874 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:13.874 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:13.874 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:13.874 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 769 2024-04-19T02:04:13.874 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.874 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:13.874 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.874 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:13.874 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102479+0000", 2024-04-19T02:04:13.874 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.874 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.874 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.874 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:13.875 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102479+0000", 2024-04-19T02:04:13.875 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.875 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.875 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.875 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:13.875 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102481+0000", 2024-04-19T02:04:13.875 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.843e-06 2024-04-19T02:04:13.875 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.875 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.875 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:13.875 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102492+0000", 2024-04-19T02:04:13.875 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0793e-05 2024-04-19T02:04:13.875 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.875 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.875 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:13.876 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102516+0000", 2024-04-19T02:04:13.876 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3873e-05 2024-04-19T02:04:13.876 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.876 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.876 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:13.876 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102532+0000", 2024-04-19T02:04:13.876 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6674999999999999e-05 2024-04-19T02:04:13.876 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.876 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.876 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:13.876 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.123043+0000", 2024-04-19T02:04:13.876 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020510626000000001 2024-04-19T02:04:13.876 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.876 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.876 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:13.877 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.123345+0000", 2024-04-19T02:04:13.877 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000302202 2024-04-19T02:04:13.877 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.877 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.877 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:13.877 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.124785+0000", 2024-04-19T02:04:13.877 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001439596 2024-04-19T02:04:13.877 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.877 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.877 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:13.877 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.127574+0000", 2024-04-19T02:04:13.877 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002789822 2024-04-19T02:04:13.877 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.877 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.878 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:04:13.878 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.134878+0000", 2024-04-19T02:04:13.878 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0073030400000000002 2024-04-19T02:04:13.878 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.878 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.878 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:13.878 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.135025+0000", 2024-04-19T02:04:13.878 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014709099999999999 2024-04-19T02:04:13.878 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.878 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:13.878 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.878 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.878 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.878 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:770 3.5s0 3:a4ae10cb:::smithi05930959-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e67)", 2024-04-19T02:04:13.879 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:51.102554+0000", 2024-04-19T02:04:13.879 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 22.712538740999999, 2024-04-19T02:04:13.879 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.037176628000000003, 2024-04-19T02:04:13.879 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:13.879 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:13.879 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:04:13.879 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:13.879 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:13.879 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:13.879 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 770 2024-04-19T02:04:13.879 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.879 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:13.879 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.879 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:13.879 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102554+0000", 2024-04-19T02:04:13.880 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.880 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.880 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.880 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:13.880 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102554+0000", 2024-04-19T02:04:13.880 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.880 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.880 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.880 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:13.880 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102555+0000", 2024-04-19T02:04:13.880 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1799999999999999e-06 2024-04-19T02:04:13.880 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.880 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.880 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:13.880 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102562+0000", 2024-04-19T02:04:13.881 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5039999999999999e-06 2024-04-19T02:04:13.881 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.881 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.881 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:13.881 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102585+0000", 2024-04-19T02:04:13.881 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3159e-05 2024-04-19T02:04:13.881 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.881 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.881 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:13.881 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.102602+0000", 2024-04-19T02:04:13.881 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6892e-05 2024-04-19T02:04:13.881 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.881 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.881 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:13.881 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.127057+0000", 2024-04-19T02:04:13.882 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0244551 2024-04-19T02:04:13.882 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.882 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.882 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:04:13.882 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.127389+0000", 2024-04-19T02:04:13.882 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00033189999999999999 2024-04-19T02:04:13.882 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.882 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.882 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:13.882 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.136002+0000", 2024-04-19T02:04:13.882 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0086132810000000004 2024-04-19T02:04:13.882 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.882 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.882 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:13.882 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.136156+0000", 2024-04-19T02:04:13.883 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015367399999999999 2024-04-19T02:04:13.883 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.883 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.883 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:13.883 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:51.139731+0000", 2024-04-19T02:04:13.883 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0035749380000000002 2024-04-19T02:04:13.883 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.883 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:13.883 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.883 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.883 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.883 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:855 3.5s0 3:a4ae10cb:::smithi05930959-29:head [rollback 19] snapc 1c=[1c,1b,19,17,16,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-04-19T02:04:13.883 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:59.117957+0000", 2024-04-19T02:04:13.883 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 14.697135989, 2024-04-19T02:04:13.884 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.023098585000000001, 2024-04-19T02:04:13.884 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:13.884 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:13.884 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:13.884 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:13.884 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:13.884 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:13.884 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 855 2024-04-19T02:04:13.884 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.884 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:13.884 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.884 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:13.884 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:59.117957+0000", 2024-04-19T02:04:13.884 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.884 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.885 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.885 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:13.885 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:59.117957+0000", 2024-04-19T02:04:13.885 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.885 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.885 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.885 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:13.885 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:59.117959+0000", 2024-04-19T02:04:13.885 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.587e-06 2024-04-19T02:04:13.885 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.885 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.885 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:13.885 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:59.117967+0000", 2024-04-19T02:04:13.885 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2269999999999993e-06 2024-04-19T02:04:13.885 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.885 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.886 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:13.886 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:59.117989+0000", 2024-04-19T02:04:13.886 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2225000000000002e-05 2024-04-19T02:04:13.886 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.886 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.886 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:13.886 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:59.118006+0000", 2024-04-19T02:04:13.886 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6857000000000001e-05 2024-04-19T02:04:13.886 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.886 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.886 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:13.886 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:59.118274+0000", 2024-04-19T02:04:13.886 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00026768799999999998 2024-04-19T02:04:13.886 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.886 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.887 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:04:13.887 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:59.118294+0000", 2024-04-19T02:04:13.887 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0262e-05 2024-04-19T02:04:13.887 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.887 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.887 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:13.887 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:59.130695+0000", 2024-04-19T02:04:13.887 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.01240138 2024-04-19T02:04:13.887 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.887 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.887 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:13.887 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:59.130981+0000", 2024-04-19T02:04:13.887 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028589799999999998 2024-04-19T02:04:13.887 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.887 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.888 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:13.888 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:59.132847+0000", 2024-04-19T02:04:13.888 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001866298 2024-04-19T02:04:13.888 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.888 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.888 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:13.888 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:59.139207+0000", 2024-04-19T02:04:13.888 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0063595309999999999 2024-04-19T02:04:13.888 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.888 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.888 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:04:13.888 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:59.140878+0000", 2024-04-19T02:04:13.888 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0016710010000000001 2024-04-19T02:04:13.888 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.888 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.889 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:13.889 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:59.141056+0000", 2024-04-19T02:04:13.889 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017763100000000001 2024-04-19T02:04:13.889 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.889 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:13.889 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.889 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.889 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.889 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:856 3.5s0 3:a4ae10cb:::smithi05930959-29:head [append 0~0] snapc 1c=[1c,1b,19,17,16,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-04-19T02:04:13.889 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:59.118028+0000", 2024-04-19T02:04:13.889 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 14.697064541, 2024-04-19T02:04:13.889 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.023423950999999998, 2024-04-19T02:04:13.889 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:13.889 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:13.889 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:13.889 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:13.890 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:13.890 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:13.890 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 856 2024-04-19T02:04:13.890 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.890 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:13.890 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.890 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:13.890 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:59.118028+0000", 2024-04-19T02:04:13.890 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.890 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.890 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.890 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:13.890 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:59.118028+0000", 2024-04-19T02:04:13.890 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.891 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.891 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.891 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:13.891 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:59.118030+0000", 2024-04-19T02:04:13.891 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.747e-06 2024-04-19T02:04:13.891 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.891 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.891 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:13.891 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:59.118036+0000", 2024-04-19T02:04:13.891 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1970000000000001e-06 2024-04-19T02:04:13.891 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.891 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.891 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:13.891 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:59.118054+0000", 2024-04-19T02:04:13.891 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7759000000000001e-05 2024-04-19T02:04:13.892 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.892 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.892 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:13.892 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:59.118069+0000", 2024-04-19T02:04:13.892 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4613000000000001e-05 2024-04-19T02:04:13.892 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.892 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.892 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:13.892 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:59.118373+0000", 2024-04-19T02:04:13.892 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00030470300000000003 2024-04-19T02:04:13.892 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.892 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.892 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:04:13.892 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:59.118391+0000", 2024-04-19T02:04:13.892 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7655999999999998e-05 2024-04-19T02:04:13.893 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.893 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.893 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:13.893 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:59.136179+0000", 2024-04-19T02:04:13.893 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017788335999999998 2024-04-19T02:04:13.893 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.893 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.893 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:13.893 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:59.136615+0000", 2024-04-19T02:04:13.893 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00043525299999999997 2024-04-19T02:04:13.893 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.893 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.893 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:13.893 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:59.138223+0000", 2024-04-19T02:04:13.893 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0016086550000000001 2024-04-19T02:04:13.893 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.894 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.894 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:13.894 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:59.140654+0000", 2024-04-19T02:04:13.894 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002431047 2024-04-19T02:04:13.894 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.894 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.894 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:04:13.894 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:59.141342+0000", 2024-04-19T02:04:13.894 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000687902 2024-04-19T02:04:13.894 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.894 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.894 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:13.894 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:59.141452+0000", 2024-04-19T02:04:13.894 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000110083 2024-04-19T02:04:13.895 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.895 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:13.895 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.895 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.895 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.895 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.1s4 80/26 ECSubWrite(tid=79, reqid=osd.0.0:79, at_version=80'45, trim_to=0'0, roll_forward_to=79'43))", 2024-04-19T02:04:13.895 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:04:04.159002+0000", 2024-04-19T02:04:13.895 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.6560912620000003, 2024-04-19T02:04:13.895 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.023019212000000001, 2024-04-19T02:04:13.895 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:13.895 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:13.895 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:04:13.895 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:13.895 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.895 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:13.896 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.159002+0000", 2024-04-19T02:04:13.896 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.896 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.896 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.896 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:13.896 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.159002+0000", 2024-04-19T02:04:13.896 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.896 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.896 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.896 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:13.896 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.159003+0000", 2024-04-19T02:04:13.896 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.373e-06 2024-04-19T02:04:13.896 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.896 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.896 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:13.897 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.159087+0000", 2024-04-19T02:04:13.897 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3547999999999994e-05 2024-04-19T02:04:13.897 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.897 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.897 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:13.897 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.159111+0000", 2024-04-19T02:04:13.897 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4822000000000001e-05 2024-04-19T02:04:13.897 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.897 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.897 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:13.897 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.159134+0000", 2024-04-19T02:04:13.897 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2362999999999999e-05 2024-04-19T02:04:13.897 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.897 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.897 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:13.898 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.159288+0000", 2024-04-19T02:04:13.898 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000154122 2024-04-19T02:04:13.898 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.898 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.898 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:13.898 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.159314+0000", 2024-04-19T02:04:13.898 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6366000000000001e-05 2024-04-19T02:04:13.898 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.898 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.898 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:13.898 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.181914+0000", 2024-04-19T02:04:13.898 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022600141000000001 2024-04-19T02:04:13.898 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.898 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.898 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:13.899 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.182021+0000", 2024-04-19T02:04:13.899 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000106477 2024-04-19T02:04:13.899 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.899 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:13.899 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.899 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.899 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.899 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.1s4 80/26 ECSubWrite(tid=80, reqid=osd.0.0:80, at_version=80'47, trim_to=0'0, roll_forward_to=79'43))", 2024-04-19T02:04:13.899 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:04:04.159185+0000", 2024-04-19T02:04:13.899 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.655908191, 2024-04-19T02:04:13.899 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.031655100999999998, 2024-04-19T02:04:13.899 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:13.899 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:13.899 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:04:13.899 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:13.900 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.900 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:13.900 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.159185+0000", 2024-04-19T02:04:13.900 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.900 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.900 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.900 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:13.900 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.159185+0000", 2024-04-19T02:04:13.900 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.900 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.900 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.900 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:13.900 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.159186+0000", 2024-04-19T02:04:13.900 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9999999999999996e-07 2024-04-19T02:04:13.900 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.901 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.901 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:13.901 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.159229+0000", 2024-04-19T02:04:13.901 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3132e-05 2024-04-19T02:04:13.901 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.901 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.901 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:13.901 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.159250+0000", 2024-04-19T02:04:13.901 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0890000000000002e-05 2024-04-19T02:04:13.901 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.901 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.901 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:13.901 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.159267+0000", 2024-04-19T02:04:13.901 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7773000000000001e-05 2024-04-19T02:04:13.901 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.902 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.902 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:13.902 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.167484+0000", 2024-04-19T02:04:13.902 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0082163400000000008 2024-04-19T02:04:13.902 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.902 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.902 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:13.902 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.167510+0000", 2024-04-19T02:04:13.902 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6386000000000001e-05 2024-04-19T02:04:13.902 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.902 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.902 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:13.902 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.190741+0000", 2024-04-19T02:04:13.902 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.023231213000000001 2024-04-19T02:04:13.903 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.903 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.903 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:13.903 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.190840+0000", 2024-04-19T02:04:13.903 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8566999999999993e-05 2024-04-19T02:04:13.903 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.903 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:13.903 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.903 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.903 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.903 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.bs5 80/26 ECSubWrite(tid=353, reqid=osd.3.0:353, at_version=80'79, trim_to=0'0, roll_forward_to=79'75))", 2024-04-19T02:04:13.903 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:04:04.169920+0000", 2024-04-19T02:04:13.903 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.6451731719999998, 2024-04-19T02:04:13.903 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.030160233000000002, 2024-04-19T02:04:13.903 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:13.904 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:13.904 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:04:13.904 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:13.904 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.904 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:13.904 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.169920+0000", 2024-04-19T02:04:13.904 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.904 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.904 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.904 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:13.904 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.169920+0000", 2024-04-19T02:04:13.904 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.904 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.904 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.904 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:13.904 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.169921+0000", 2024-04-19T02:04:13.905 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.006e-06 2024-04-19T02:04:13.905 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.905 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.905 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:13.905 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.169978+0000", 2024-04-19T02:04:13.905 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7552000000000001e-05 2024-04-19T02:04:13.905 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.905 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.905 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:13.905 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.170001+0000", 2024-04-19T02:04:13.905 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2955999999999998e-05 2024-04-19T02:04:13.905 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.905 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.905 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:13.905 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.170021+0000", 2024-04-19T02:04:13.906 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9335999999999999e-05 2024-04-19T02:04:13.906 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.906 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.906 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:13.906 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.183740+0000", 2024-04-19T02:04:13.906 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013719576000000001 2024-04-19T02:04:13.906 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.906 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.906 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:13.906 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.183759+0000", 2024-04-19T02:04:13.906 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.925e-05 2024-04-19T02:04:13.906 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.906 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.906 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:13.907 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.200007+0000", 2024-04-19T02:04:13.907 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016247686000000001 2024-04-19T02:04:13.907 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.907 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.907 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:13.907 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.200080+0000", 2024-04-19T02:04:13.907 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2871000000000001e-05 2024-04-19T02:04:13.907 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.907 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:13.907 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.907 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.907 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.907 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.4s2 80/26 ECSubWrite(tid=140, reqid=osd.1.0:140, at_version=80'125, trim_to=0'0, roll_forward_to=80'121))", 2024-04-19T02:04:13.907 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:04:04.172777+0000", 2024-04-19T02:04:13.907 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.642315687, 2024-04-19T02:04:13.907 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.033629829999999999, 2024-04-19T02:04:13.908 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:13.908 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:13.908 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:04:13.908 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:13.908 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.908 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:13.908 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.172777+0000", 2024-04-19T02:04:13.908 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.908 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.908 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.908 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:13.908 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.172777+0000", 2024-04-19T02:04:13.908 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:13.908 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.909 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.909 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:13.909 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.172779+0000", 2024-04-19T02:04:13.909 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.95e-06 2024-04-19T02:04:13.909 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.909 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.909 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:13.909 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.172828+0000", 2024-04-19T02:04:13.909 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8307999999999999e-05 2024-04-19T02:04:13.909 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.909 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.909 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:13.909 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.172872+0000", 2024-04-19T02:04:13.909 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4245000000000001e-05 2024-04-19T02:04:13.909 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.910 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.910 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:13.910 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.173041+0000", 2024-04-19T02:04:13.910 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016961499999999999 2024-04-19T02:04:13.910 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.910 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.910 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:13.910 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.182079+0000", 2024-04-19T02:04:13.910 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0090373380000000007 2024-04-19T02:04:13.910 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.910 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.910 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:13.910 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.182107+0000", 2024-04-19T02:04:13.910 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8722e-05 2024-04-19T02:04:13.911 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.911 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.911 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:13.911 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.206324+0000", 2024-04-19T02:04:13.911 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.024216393999999999 2024-04-19T02:04:13.911 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:13.911 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:13.911 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:13.911 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.206407+0000", 2024-04-19T02:04:13.911 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3257999999999995e-05 2024-04-19T02:04:13.911 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.911 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:13.911 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.911 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:13.911 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:13.912 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:04:13.913 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-19T02:04:13.914 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:04:13.912+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:04:14.014 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-04-19T02:04:14.016+0000 7f887acbb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25035) UID: 0 2024-04-19T02:04:14.042 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:04:14.042 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:04:14.042 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-04-19T02:04:14.042 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:04:14.057 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-19T02:04:14.084 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'unique_pool_0' removed 2024-04-19T02:04:14.084 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-19T02:04:14.115 INFO:tasks.ceph.osd.10.smithi179.stderr:2024-04-19T02:04:14.116+0000 7f11c4851640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25309) UID: 0 2024-04-19T02:04:14.117 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-04-19T02:04:14.138 INFO:tasks.thrashosds:joining thrashosds 2024-04-19T02:04:14.184 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:04:14.184 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-04-19T02:04:14.184 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-04-19T02:04:14.184 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-04-19T02:04:14.184 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:04:14.198 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-19T02:04:14.371 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-19T02:04:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-04-19T02:04:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-04-19T02:04:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-04-19T02:04:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/26 ECSubWrite(tid=134, reqid=osd.3.0:134, at_version=42'65, trim_to=0'0, roll_forward_to=40'61))", 2024-04-19T02:04:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.751576+0000", 2024-04-19T02:04:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 47.619529061999998, 2024-04-19T02:04:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.027269287, 2024-04-19T02:04:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:04:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751576+0000", 2024-04-19T02:04:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751576+0000", 2024-04-19T02:04:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751578+0000", 2024-04-19T02:04:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9659999999999999e-06 2024-04-19T02:04:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.751651+0000", 2024-04-19T02:04:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2781000000000007e-05 2024-04-19T02:04:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755528+0000", 2024-04-19T02:04:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0038770359999999999 2024-04-19T02:04:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755558+0000", 2024-04-19T02:04:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9414e-05 2024-04-19T02:04:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755853+0000", 2024-04-19T02:04:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00029508699999999998 2024-04-19T02:04:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.755881+0000", 2024-04-19T02:04:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7965000000000001e-05 2024-04-19T02:04:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778764+0000", 2024-04-19T02:04:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022883244000000001 2024-04-19T02:04:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.778846+0000", 2024-04-19T02:04:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1793999999999996e-05 2024-04-19T02:04:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:371 3.as0 3:54128c7a:::smithi05930959-20:head [read 0~1605632,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:04:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.760138+0000", 2024-04-19T02:04:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 47.610967631000001, 2024-04-19T02:04:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.031633367000000003, 2024-04-19T02:04:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:04:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 371 2024-04-19T02:04:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760138+0000", 2024-04-19T02:04:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.378 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:14.378 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760138+0000", 2024-04-19T02:04:14.378 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.378 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.378 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.378 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:14.378 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760141+0000", 2024-04-19T02:04:14.378 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0790000000000001e-06 2024-04-19T02:04:14.378 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.378 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.378 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:14.378 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760155+0000", 2024-04-19T02:04:14.378 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4083000000000001e-05 2024-04-19T02:04:14.378 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.378 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760187+0000", 2024-04-19T02:04:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1964000000000001e-05 2024-04-19T02:04:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760208+0000", 2024-04-19T02:04:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1395000000000001e-05 2024-04-19T02:04:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760524+0000", 2024-04-19T02:04:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00031564199999999998 2024-04-19T02:04:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:04:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760793+0000", 2024-04-19T02:04:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000268819 2024-04-19T02:04:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774492+0000", 2024-04-19T02:04:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013699065999999999 2024-04-19T02:04:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.774730+0000", 2024-04-19T02:04:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00023788499999999999 2024-04-19T02:04:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.791771+0000", 2024-04-19T02:04:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017041434000000001 2024-04-19T02:04:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:372 3.as0 3:54128c7a:::smithi05930959-20:head [read 0~1605632,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:04:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.760260+0000", 2024-04-19T02:04:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 47.610845263999998, 2024-04-19T02:04:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.034139714000000002, 2024-04-19T02:04:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:04:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 372 2024-04-19T02:04:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760260+0000", 2024-04-19T02:04:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760260+0000", 2024-04-19T02:04:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760262+0000", 2024-04-19T02:04:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6029999999999999e-06 2024-04-19T02:04:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760272+0000", 2024-04-19T02:04:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6560000000000004e-06 2024-04-19T02:04:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760298+0000", 2024-04-19T02:04:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6625e-05 2024-04-19T02:04:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760316+0000", 2024-04-19T02:04:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7459e-05 2024-04-19T02:04:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760885+0000", 2024-04-19T02:04:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00056941200000000004 2024-04-19T02:04:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:04:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761126+0000", 2024-04-19T02:04:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000241124 2024-04-19T02:04:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.775702+0000", 2024-04-19T02:04:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014575764 2024-04-19T02:04:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.777674+0000", 2024-04-19T02:04:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001972297 2024-04-19T02:04:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.794400+0000", 2024-04-19T02:04:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016725773999999999 2024-04-19T02:04:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:373 3.as0 3:54128c7a:::smithi05930959-20:head [read 0~1605632,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-19T02:04:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:26.760341+0000", 2024-04-19T02:04:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 47.610764809999999, 2024-04-19T02:04:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.043665984999999997, 2024-04-19T02:04:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:04:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 373 2024-04-19T02:04:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760341+0000", 2024-04-19T02:04:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760341+0000", 2024-04-19T02:04:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760342+0000", 2024-04-19T02:04:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.497e-06 2024-04-19T02:04:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760350+0000", 2024-04-19T02:04:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1119999999999996e-06 2024-04-19T02:04:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760370+0000", 2024-04-19T02:04:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9219e-05 2024-04-19T02:04:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.760385+0000", 2024-04-19T02:04:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5905000000000001e-05 2024-04-19T02:04:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761214+0000", 2024-04-19T02:04:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000828939 2024-04-19T02:04:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:04:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.761439+0000", 2024-04-19T02:04:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022493900000000001 2024-04-19T02:04:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.794533+0000", 2024-04-19T02:04:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.033094077999999999 2024-04-19T02:04:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.794696+0000", 2024-04-19T02:04:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000162442 2024-04-19T02:04:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:26.804007+0000", 2024-04-19T02:04:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0093108540000000004 2024-04-19T02:04:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:381 3.as0 3:512144ec:::smithi05930959-48:head [create,setxattr pnc]aaY (11788) in=7b,setxattr ZTVVStUpk__YaS\\blR\\XgesYT[Or[[aUett_nil] (1859) in=40b,setxattr ZtotWOVcPP_oiTUUTlqtYZsNneQnmhc\\bNdNpsgo (1402) in=40b,setxattr cR\\akUitgYfao[oisZ^ach\\i_kdge\\[c_mbmUf]^ (313) in=40b,setxattr pRe[UcWhXR^oagaSYScYPQa]r[hdldO\\S`V^[nh^ (874) in=40b,setxattr srVXOoW^mbi\\ccUpc`barbsc (11788) in=24b,setxattr pqjhUkafUjRqdpObgi_ZX^QV (11788) in=24b,setxattr q]Of\\bd_VgdoaP\\k^tmmpUqq^otrVVOdrtgU]nVk (12280) in=40b,setxattr ]Ple__\\nrXPQYecmQXf_o^bgN`mofssSPQrYslSO (1859) in=40b,setxattr ZiQcjT`UZQ]es__dmo^h]NdV^`qT_hniQtVY`WsT (770) in=40b,setxattr gpT^oWmeYZpbVdojjWnUWO[YfZptiQhdWdhs]Zo^ (841) in=40b,setxattr `Zgeqf\\Rb][aUdTVXYhj[ealjcUnRaiY (11788) in=32b,setxattr [dN]aooYVPYS^Q`_`bqSNVSS^`ZT\\csYZSrnjtUT (11296) in=40b,setxattr ehcNktRfRjqWWi[^XWcZokN]Wb[YnkeNl\\lkt_t\\ (1262) in=40b,setxattr OdsZcnit[de]qNV (11788) in=15b,setxattr ^]^lenZo\\TjoZOpONRno]jtaWOd`tqNdWcYccWNk (560) in=40b,setxattr ZUohXUqoPmYaaTceWo\\SRigth\\qP[Tg_^^asjknX (1085) in=40b,setxattr lR`dUarlc[n`ctj^TlY\\QpagN_doPfV`lZQ^kNtY (526) in=40b,setxattr [iXNTskqfmN\\qVhqrm`rdXpaN_VaeRbd^O\\Z_gcY (1261) in=40b,setxattr t_jR_akNV_aXVR^icaga]cP]baNne_f_UdsoeVUk (1894) in=40b] snapc 9=[8,6] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-04-19T02:04:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:27.087092+0000", 2024-04-19T02:04:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 47.284013815999998, 2024-04-19T02:04:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.025861967999999999, 2024-04-19T02:04:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 381 2024-04-19T02:04:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087092+0000", 2024-04-19T02:04:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087092+0000", 2024-04-19T02:04:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087095+0000", 2024-04-19T02:04:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2799999999999999e-06 2024-04-19T02:04:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087306+0000", 2024-04-19T02:04:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021136000000000001 2024-04-19T02:04:14.394 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.394 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.394 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:14.394 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087341+0000", 2024-04-19T02:04:14.394 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4654999999999999e-05 2024-04-19T02:04:14.394 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.394 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.394 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:14.394 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087358+0000", 2024-04-19T02:04:14.394 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7261e-05 2024-04-19T02:04:14.394 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.394 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.394 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.394 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087536+0000", 2024-04-19T02:04:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000177424 2024-04-19T02:04:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op must wait for map", 2024-04-19T02:04:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.087551+0000", 2024-04-19T02:04:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5362e-05 2024-04-19T02:04:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.092762+0000", 2024-04-19T02:04:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0052113589999999996 2024-04-19T02:04:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.092952+0000", 2024-04-19T02:04:14.396 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00018992100000000001 2024-04-19T02:04:14.396 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.396 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.396 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:14.396 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.095255+0000", 2024-04-19T02:04:14.396 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0023027030000000001 2024-04-19T02:04:14.396 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.396 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.396 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:14.396 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.101758+0000", 2024-04-19T02:04:14.396 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0065034380000000003 2024-04-19T02:04:14.396 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.396 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.396 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:04:14.396 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.112669+0000", 2024-04-19T02:04:14.396 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010910737 2024-04-19T02:04:14.397 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.397 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.397 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:14.397 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:27.112954+0000", 2024-04-19T02:04:14.397 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028446800000000001 2024-04-19T02:04:14.397 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.397 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:14.397 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.397 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.397 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.397 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:436 3.as0 3:5a72bad2:::smithi05930959-41:head [read 0~3293184,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-04-19T02:04:14.397 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:30.194819+0000", 2024-04-19T02:04:14.397 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 44.176286513000001, 2024-04-19T02:04:14.398 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.028970988999999999, 2024-04-19T02:04:14.398 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:14.398 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:14.398 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:04:14.398 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:14.398 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:14.398 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:14.398 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 436 2024-04-19T02:04:14.398 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.398 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:14.398 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.398 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:14.398 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194819+0000", 2024-04-19T02:04:14.398 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.398 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.399 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.399 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:14.399 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194819+0000", 2024-04-19T02:04:14.399 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.399 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.399 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.399 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:14.399 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194820+0000", 2024-04-19T02:04:14.399 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1829999999999999e-06 2024-04-19T02:04:14.399 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.399 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.399 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:14.399 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194828+0000", 2024-04-19T02:04:14.399 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.306e-06 2024-04-19T02:04:14.399 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.400 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.400 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:14.400 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194846+0000", 2024-04-19T02:04:14.400 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8592000000000001e-05 2024-04-19T02:04:14.400 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.400 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.400 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:14.400 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.194860+0000", 2024-04-19T02:04:14.400 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4185999999999999e-05 2024-04-19T02:04:14.400 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.400 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.400 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.400 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198124+0000", 2024-04-19T02:04:14.400 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0032639430000000001 2024-04-19T02:04:14.400 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.198217+0000", 2024-04-19T02:04:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2303e-05 2024-04-19T02:04:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:30.223790+0000", 2024-04-19T02:04:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.025573476000000001 2024-04-19T02:04:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:502 3.fs0 3:fbe22738:::smithi05930959-7:b [read 0~2883584,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-04-19T02:04:14.402 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:32.048354+0000", 2024-04-19T02:04:14.402 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 42.322751799999999, 2024-04-19T02:04:14.402 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.035084603999999998, 2024-04-19T02:04:14.402 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:14.402 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:14.402 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:04:14.402 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:14.402 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:14.402 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:14.402 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 502 2024-04-19T02:04:14.402 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.402 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:14.402 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.402 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:14.403 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048354+0000", 2024-04-19T02:04:14.403 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.403 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.403 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.403 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:14.403 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048354+0000", 2024-04-19T02:04:14.403 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.403 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.403 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.403 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:14.403 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048355+0000", 2024-04-19T02:04:14.403 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1659999999999999e-06 2024-04-19T02:04:14.403 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.403 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.403 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:14.404 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048364+0000", 2024-04-19T02:04:14.404 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5829999999999996e-06 2024-04-19T02:04:14.404 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.404 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.404 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:14.404 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048383+0000", 2024-04-19T02:04:14.404 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8108999999999999e-05 2024-04-19T02:04:14.404 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.404 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.404 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:14.404 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048400+0000", 2024-04-19T02:04:14.404 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7051999999999999e-05 2024-04-19T02:04:14.404 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.404 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.404 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.405 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048713+0000", 2024-04-19T02:04:14.405 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00031348900000000002 2024-04-19T02:04:14.405 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.405 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.405 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:04:14.405 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.048723+0000", 2024-04-19T02:04:14.405 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6760000000000002e-06 2024-04-19T02:04:14.405 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.405 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.405 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.405 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.056039+0000", 2024-04-19T02:04:14.405 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0073158099999999998 2024-04-19T02:04:14.405 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.405 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.406 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:14.406 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.056328+0000", 2024-04-19T02:04:14.406 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028893700000000002 2024-04-19T02:04:14.406 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.406 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.406 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:14.406 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:32.083438+0000", 2024-04-19T02:04:14.406 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.027110782 2024-04-19T02:04:14.406 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.406 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:14.406 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.406 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.406 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.406 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:535 3.fs0 3:f8a4544e:::smithi05930959-1:head [copy-from ver 31] snapc e=[e,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-19T02:04:14.407 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:36.137521+0000", 2024-04-19T02:04:14.407 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 38.23358477, 2024-04-19T02:04:14.407 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.030940776999999999, 2024-04-19T02:04:14.407 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:14.407 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:14.407 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:14.407 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:14.407 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:14.407 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:14.407 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 535 2024-04-19T02:04:14.407 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.407 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:14.407 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.407 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:14.408 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137521+0000", 2024-04-19T02:04:14.408 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.408 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.408 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.408 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:14.408 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137521+0000", 2024-04-19T02:04:14.408 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.408 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.408 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.408 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:14.408 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137524+0000", 2024-04-19T02:04:14.408 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3969999999999998e-06 2024-04-19T02:04:14.408 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.408 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.408 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:14.409 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137540+0000", 2024-04-19T02:04:14.409 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5619000000000001e-05 2024-04-19T02:04:14.409 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.409 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.409 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:14.409 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137571+0000", 2024-04-19T02:04:14.409 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0901000000000001e-05 2024-04-19T02:04:14.409 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.409 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.409 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:14.409 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137591+0000", 2024-04-19T02:04:14.409 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0279e-05 2024-04-19T02:04:14.409 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.409 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.409 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.409 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137849+0000", 2024-04-19T02:04:14.410 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000258535 2024-04-19T02:04:14.410 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.410 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.410 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op must wait for map", 2024-04-19T02:04:14.410 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137872+0000", 2024-04-19T02:04:14.410 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2507999999999999e-05 2024-04-19T02:04:14.410 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.410 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.410 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.410 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.141172+0000", 2024-04-19T02:04:14.410 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003300234 2024-04-19T02:04:14.410 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.410 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.410 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:14.411 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.141447+0000", 2024-04-19T02:04:14.411 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00027449700000000001 2024-04-19T02:04:14.411 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.411 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.411 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:14.411 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.161979+0000", 2024-04-19T02:04:14.411 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020531964 2024-04-19T02:04:14.411 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.411 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.411 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:14.411 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.165458+0000", 2024-04-19T02:04:14.411 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0034790060000000002 2024-04-19T02:04:14.411 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.412 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.412 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:04:14.412 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.168324+0000", 2024-04-19T02:04:14.412 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0028665280000000001 2024-04-19T02:04:14.412 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.412 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.412 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:14.412 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.168462+0000", 2024-04-19T02:04:14.412 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013730900000000001 2024-04-19T02:04:14.412 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.412 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:14.412 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.412 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.412 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.412 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:536 3.fs0 3:f8a4544e:::smithi05930959-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-04-19T02:04:14.413 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:36.137666+0000", 2024-04-19T02:04:14.413 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 38.233439582000003, 2024-04-19T02:04:14.413 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.027514961000000001, 2024-04-19T02:04:14.413 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:14.413 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:14.413 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:04:14.413 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:14.413 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:14.413 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:14.413 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 536 2024-04-19T02:04:14.413 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.413 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:14.413 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.413 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:14.413 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137666+0000", 2024-04-19T02:04:14.414 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.414 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.414 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.414 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:14.414 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137666+0000", 2024-04-19T02:04:14.414 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.414 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.414 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.414 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:14.414 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137668+0000", 2024-04-19T02:04:14.414 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9199999999999998e-06 2024-04-19T02:04:14.414 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.414 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.414 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:14.415 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137675+0000", 2024-04-19T02:04:14.415 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.153e-06 2024-04-19T02:04:14.415 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.415 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.415 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:14.415 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137697+0000", 2024-04-19T02:04:14.415 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1522000000000002e-05 2024-04-19T02:04:14.415 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.415 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.415 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:14.415 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.137713+0000", 2024-04-19T02:04:14.415 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6016e-05 2024-04-19T02:04:14.415 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.415 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.415 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.415 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.138006+0000", 2024-04-19T02:04:14.416 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00029325899999999998 2024-04-19T02:04:14.416 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.416 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.416 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:04:14.416 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.138022+0000", 2024-04-19T02:04:14.416 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6563e-05 2024-04-19T02:04:14.416 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.416 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.416 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.416 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.141952+0000", 2024-04-19T02:04:14.416 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039293360000000003 2024-04-19T02:04:14.416 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.416 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.416 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for blocked object", 2024-04-19T02:04:14.417 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.142198+0000", 2024-04-19T02:04:14.417 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00024626500000000001 2024-04-19T02:04:14.417 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.417 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.417 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.417 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.164631+0000", 2024-04-19T02:04:14.417 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022432688999999999 2024-04-19T02:04:14.417 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.417 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.417 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:14.417 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.164884+0000", 2024-04-19T02:04:14.417 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000252892 2024-04-19T02:04:14.417 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.417 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.418 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:14.418 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:36.165181+0000", 2024-04-19T02:04:14.418 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00029734599999999998 2024-04-19T02:04:14.418 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.418 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:14.418 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.418 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.418 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.418 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:788 3.as0 3:5a72bad2:::smithi05930959-41:head [read 0~7405568,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-04-19T02:04:14.418 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:54.059369+0000", 2024-04-19T02:04:14.418 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 20.311736103000001, 2024-04-19T02:04:14.418 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.032465221000000002, 2024-04-19T02:04:14.418 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:14.418 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:14.419 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:04:14.419 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:14.419 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:14.419 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:14.419 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 788 2024-04-19T02:04:14.419 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.419 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:14.419 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.419 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:14.419 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059369+0000", 2024-04-19T02:04:14.419 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.419 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.419 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.419 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:14.420 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059369+0000", 2024-04-19T02:04:14.420 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.420 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.420 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.420 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:14.420 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059371+0000", 2024-04-19T02:04:14.420 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.782e-06 2024-04-19T02:04:14.420 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.420 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.420 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:14.420 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059378+0000", 2024-04-19T02:04:14.420 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2899999999999997e-06 2024-04-19T02:04:14.420 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.420 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.420 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:14.420 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059393+0000", 2024-04-19T02:04:14.421 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4792e-05 2024-04-19T02:04:14.421 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.421 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.421 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:14.421 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059403+0000", 2024-04-19T02:04:14.421 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8339999999999994e-06 2024-04-19T02:04:14.421 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.421 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.421 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.421 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059476+0000", 2024-04-19T02:04:14.421 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2659999999999999e-05 2024-04-19T02:04:14.421 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.421 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.421 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op must wait for map", 2024-04-19T02:04:14.421 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059482+0000", 2024-04-19T02:04:14.422 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5699999999999998e-06 2024-04-19T02:04:14.422 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.422 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.422 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.422 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.062469+0000", 2024-04-19T02:04:14.422 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002987015 2024-04-19T02:04:14.422 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.422 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.422 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:14.422 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.062613+0000", 2024-04-19T02:04:14.422 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014379099999999999 2024-04-19T02:04:14.422 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.422 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.422 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:14.423 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.091835+0000", 2024-04-19T02:04:14.423 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.029221487000000001 2024-04-19T02:04:14.423 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.423 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:14.423 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.423 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.423 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.423 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:789 3.as0 3:5a72bad2:::smithi05930959-41:head [read 0~7405568,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-04-19T02:04:14.423 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:54.059423+0000", 2024-04-19T02:04:14.423 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 20.311682885, 2024-04-19T02:04:14.423 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.039046966000000002, 2024-04-19T02:04:14.423 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:14.423 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:14.423 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:04:14.423 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 789 2024-04-19T02:04:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059423+0000", 2024-04-19T02:04:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059423+0000", 2024-04-19T02:04:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.425 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:14.425 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059423+0000", 2024-04-19T02:04:14.425 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7999999999999996e-07 2024-04-19T02:04:14.425 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.425 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.425 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:14.425 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059426+0000", 2024-04-19T02:04:14.425 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.275e-06 2024-04-19T02:04:14.425 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.425 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.425 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:14.425 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059433+0000", 2024-04-19T02:04:14.425 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8530000000000003e-06 2024-04-19T02:04:14.425 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.426 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.426 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:14.426 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059439+0000", 2024-04-19T02:04:14.426 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0299999999999999e-06 2024-04-19T02:04:14.426 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.426 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.426 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.426 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059570+0000", 2024-04-19T02:04:14.426 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013091000000000001 2024-04-19T02:04:14.426 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.426 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.426 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:04:14.426 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059582+0000", 2024-04-19T02:04:14.426 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1353e-05 2024-04-19T02:04:14.427 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.427 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.427 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.427 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.063219+0000", 2024-04-19T02:04:14.427 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0036374979999999999 2024-04-19T02:04:14.427 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.427 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.427 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:14.427 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.063350+0000", 2024-04-19T02:04:14.427 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000130605 2024-04-19T02:04:14.427 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.427 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.427 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:14.427 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.098470+0000", 2024-04-19T02:04:14.427 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.035119961999999998 2024-04-19T02:04:14.428 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.428 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:14.428 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.428 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.428 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.428 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:790 3.as0 3:5a72bad2:::smithi05930959-41:head [read 0~7405568,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-04-19T02:04:14.428 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:03:54.059448+0000", 2024-04-19T02:04:14.428 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 20.311657489000002, 2024-04-19T02:04:14.428 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.045622590999999997, 2024-04-19T02:04:14.428 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:14.428 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:14.428 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:04:14.428 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:14.428 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:14.429 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:14.429 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 790 2024-04-19T02:04:14.429 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.429 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:14.429 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.429 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:14.429 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059448+0000", 2024-04-19T02:04:14.429 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.429 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.429 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.429 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:14.429 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059448+0000", 2024-04-19T02:04:14.429 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.429 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.429 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.430 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:14.430 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059448+0000", 2024-04-19T02:04:14.430 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5999999999999999e-07 2024-04-19T02:04:14.430 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.430 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.430 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:14.430 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059451+0000", 2024-04-19T02:04:14.430 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4779999999999998e-06 2024-04-19T02:04:14.430 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.430 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.430 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:14.430 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059456+0000", 2024-04-19T02:04:14.430 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7639999999999996e-06 2024-04-19T02:04:14.430 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.430 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.431 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:14.431 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059459+0000", 2024-04-19T02:04:14.431 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8319999999999999e-06 2024-04-19T02:04:14.431 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.431 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.431 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.431 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059641+0000", 2024-04-19T02:04:14.431 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000181282 2024-04-19T02:04:14.431 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.431 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.431 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting_for_map not empty", 2024-04-19T02:04:14.431 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.059651+0000", 2024-04-19T02:04:14.431 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0679000000000001e-05 2024-04-19T02:04:14.431 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.432 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.432 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.432 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.068040+0000", 2024-04-19T02:04:14.432 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0083886610000000004 2024-04-19T02:04:14.432 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.432 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.432 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:14.432 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.068197+0000", 2024-04-19T02:04:14.432 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015667 2024-04-19T02:04:14.432 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.432 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.432 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:14.432 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:03:54.105071+0000", 2024-04-19T02:04:14.432 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.036873864999999999 2024-04-19T02:04:14.433 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.433 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:14.433 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.433 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.433 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.433 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:872 3.fs0 3:f25b6282:::smithi05930959-49:head [append 0~131072] snapc 1d=[1d,1c,1b,19,17,16,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-04-19T02:04:14.433 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:04:00.149842+0000", 2024-04-19T02:04:14.433 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 14.221263981, 2024-04-19T02:04:14.433 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.027347881000000001, 2024-04-19T02:04:14.433 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:14.433 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:14.433 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:14.433 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:14.433 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:14.433 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:14.434 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 872 2024-04-19T02:04:14.434 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.434 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:14.434 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.434 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:14.434 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.149842+0000", 2024-04-19T02:04:14.434 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.434 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.434 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.434 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:14.434 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.149842+0000", 2024-04-19T02:04:14.434 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.434 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.434 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.434 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:14.435 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.149845+0000", 2024-04-19T02:04:14.435 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.157e-06 2024-04-19T02:04:14.435 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.435 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.435 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:14.435 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.150224+0000", 2024-04-19T02:04:14.435 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00037930700000000001 2024-04-19T02:04:14.435 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.435 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.435 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:14.435 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.150256+0000", 2024-04-19T02:04:14.435 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1625999999999998e-05 2024-04-19T02:04:14.435 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.435 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.435 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:14.436 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.150271+0000", 2024-04-19T02:04:14.436 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5026e-05 2024-04-19T02:04:14.436 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.436 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.436 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.436 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.150417+0000", 2024-04-19T02:04:14.436 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014619200000000001 2024-04-19T02:04:14.436 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.436 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.436 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:14.436 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.150584+0000", 2024-04-19T02:04:14.436 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016688799999999999 2024-04-19T02:04:14.436 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.436 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.436 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:14.437 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.153513+0000", 2024-04-19T02:04:14.437 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0029297139999999999 2024-04-19T02:04:14.437 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.437 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.437 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:14.437 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.162868+0000", 2024-04-19T02:04:14.437 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0093548780000000005 2024-04-19T02:04:14.437 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.437 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.437 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:04:14.437 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.177008+0000", 2024-04-19T02:04:14.437 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014139311999999999 2024-04-19T02:04:14.437 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.437 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.437 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:14.438 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.177189+0000", 2024-04-19T02:04:14.438 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000181781 2024-04-19T02:04:14.438 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.438 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:14.438 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.438 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.438 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.438 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:873 3.fs0 3:f25b6282:::smithi05930959-49:head [setxattr _header (57) in=7b] snapc 1d=[1d,1c,1b,19,17,16,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-04-19T02:04:14.438 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:04:00.150309+0000", 2024-04-19T02:04:14.438 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 14.220796439000001, 2024-04-19T02:04:14.438 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.029702202, 2024-04-19T02:04:14.438 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:14.438 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:14.438 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:14.438 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 873 2024-04-19T02:04:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.150309+0000", 2024-04-19T02:04:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.150309+0000", 2024-04-19T02:04:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.440 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:14.440 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.150311+0000", 2024-04-19T02:04:14.440 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7460000000000001e-06 2024-04-19T02:04:14.440 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.440 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.440 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:14.440 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.150319+0000", 2024-04-19T02:04:14.440 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6170000000000007e-06 2024-04-19T02:04:14.440 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.440 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.440 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:14.440 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.150362+0000", 2024-04-19T02:04:14.440 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2228000000000003e-05 2024-04-19T02:04:14.440 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.440 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.441 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:14.441 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.150377+0000", 2024-04-19T02:04:14.441 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5379999999999998e-05 2024-04-19T02:04:14.441 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.441 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.441 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.441 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.160410+0000", 2024-04-19T02:04:14.441 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010032898 2024-04-19T02:04:14.441 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.441 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.441 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:14.441 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.160678+0000", 2024-04-19T02:04:14.441 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00026754799999999999 2024-04-19T02:04:14.441 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.441 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.442 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:14.442 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.162115+0000", 2024-04-19T02:04:14.442 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001437727 2024-04-19T02:04:14.442 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.442 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.442 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:14.442 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.169958+0000", 2024-04-19T02:04:14.442 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0078425839999999997 2024-04-19T02:04:14.442 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.442 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.442 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:04:14.442 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.179774+0000", 2024-04-19T02:04:14.442 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0098157130000000002 2024-04-19T02:04:14.442 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.443 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.443 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:14.443 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.180011+0000", 2024-04-19T02:04:14.443 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00023776099999999999 2024-04-19T02:04:14.443 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.443 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:14.443 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.443 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.443 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.443 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:874 3.fs0 3:f25b6282:::smithi05930959-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e76)", 2024-04-19T02:04:14.443 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:04:00.150437+0000", 2024-04-19T02:04:14.443 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 14.220668766999999, 2024-04-19T02:04:14.443 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.034986523999999998, 2024-04-19T02:04:14.443 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:14.443 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:14.443 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:04:14.444 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:14.444 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:14.444 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:14.444 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 874 2024-04-19T02:04:14.444 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.444 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:14.444 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.444 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:14.444 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.150437+0000", 2024-04-19T02:04:14.444 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.444 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.444 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.444 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:14.444 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.150437+0000", 2024-04-19T02:04:14.444 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.445 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.445 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.445 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:14.445 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.150439+0000", 2024-04-19T02:04:14.445 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0669999999999999e-06 2024-04-19T02:04:14.445 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.445 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.445 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:14.445 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.150446+0000", 2024-04-19T02:04:14.445 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6699999999999997e-06 2024-04-19T02:04:14.445 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.445 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.445 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:14.445 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.150466+0000", 2024-04-19T02:04:14.445 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0499000000000001e-05 2024-04-19T02:04:14.445 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.446 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.446 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:14.446 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.150480+0000", 2024-04-19T02:04:14.446 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3852999999999999e-05 2024-04-19T02:04:14.446 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.446 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.446 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.446 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.163611+0000", 2024-04-19T02:04:14.446 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013130816 2024-04-19T02:04:14.446 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.446 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.446 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-04-19T02:04:14.446 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.163906+0000", 2024-04-19T02:04:14.446 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00029571300000000002 2024-04-19T02:04:14.447 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.447 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.447 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.447 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.180874+0000", 2024-04-19T02:04:14.447 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016967697 2024-04-19T02:04:14.447 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.447 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.447 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:14.447 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.181101+0000", 2024-04-19T02:04:14.447 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022715800000000001 2024-04-19T02:04:14.447 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.447 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.447 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:14.447 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.185423+0000", 2024-04-19T02:04:14.447 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0043220510000000004 2024-04-19T02:04:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:884 3.as0 3:5a72bad2:::smithi05930959-41:head [copy-from ver 73] snapc 1d=[1d,1c,1b,19,17,16,12,10,c] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-04-19T02:04:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:04:00.150660+0000", 2024-04-19T02:04:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 14.220445864, 2024-04-19T02:04:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.077919129000000004, 2024-04-19T02:04:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-19T02:04:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:14.449 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 884 2024-04-19T02:04:14.449 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.449 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:14.449 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.449 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:14.449 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.150660+0000", 2024-04-19T02:04:14.449 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.449 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.449 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.449 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:14.449 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.150660+0000", 2024-04-19T02:04:14.449 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.449 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.449 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.449 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:14.450 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.150660+0000", 2024-04-19T02:04:14.450 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7999999999999995e-07 2024-04-19T02:04:14.450 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.450 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.450 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:14.450 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.150665+0000", 2024-04-19T02:04:14.450 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2969999999999997e-06 2024-04-19T02:04:14.450 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.450 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.450 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:14.450 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.150674+0000", 2024-04-19T02:04:14.450 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4320000000000005e-06 2024-04-19T02:04:14.450 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.450 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.451 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:14.451 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.150681+0000", 2024-04-19T02:04:14.451 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4039999999999998e-06 2024-04-19T02:04:14.451 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.451 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.451 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.451 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.172040+0000", 2024-04-19T02:04:14.451 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021358978000000001 2024-04-19T02:04:14.451 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.451 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.451 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:14.451 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.172316+0000", 2024-04-19T02:04:14.451 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00027555899999999998 2024-04-19T02:04:14.451 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.451 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.452 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:14.452 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.217314+0000", 2024-04-19T02:04:14.452 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.044997510999999997 2024-04-19T02:04:14.452 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.452 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.452 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:14.452 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.223052+0000", 2024-04-19T02:04:14.452 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0057387979999999998 2024-04-19T02:04:14.452 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.452 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.452 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-04-19T02:04:14.452 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.228492+0000", 2024-04-19T02:04:14.452 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0054395809999999998 2024-04-19T02:04:14.452 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.453 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.453 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:14.453 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.228579+0000", 2024-04-19T02:04:14.453 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6889000000000001e-05 2024-04-19T02:04:14.453 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.453 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:14.453 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.453 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.453 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.453 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.4509.0:885 3.as0 3:5a72bad2:::smithi05930959-41:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e76)", 2024-04-19T02:04:14.453 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:04:00.150694+0000", 2024-04-19T02:04:14.453 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 14.220411688, 2024-04-19T02:04:14.453 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.072349948999999997, 2024-04-19T02:04:14.453 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:14.453 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:14.454 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-04-19T02:04:14.454 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-04-19T02:04:14.454 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.4509", 2024-04-19T02:04:14.454 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.59:0/1982828923", 2024-04-19T02:04:14.454 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 885 2024-04-19T02:04:14.454 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.454 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:14.454 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.454 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:14.454 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.150694+0000", 2024-04-19T02:04:14.454 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.454 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.454 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.455 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:14.455 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.150694+0000", 2024-04-19T02:04:14.455 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.455 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.455 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.455 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:14.455 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.150695+0000", 2024-04-19T02:04:14.455 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4000000000000001e-07 2024-04-19T02:04:14.455 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.455 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.455 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:14.455 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.150711+0000", 2024-04-19T02:04:14.455 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6562000000000002e-05 2024-04-19T02:04:14.455 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.456 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.456 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:14.456 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.150725+0000", 2024-04-19T02:04:14.456 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3499999999999999e-05 2024-04-19T02:04:14.456 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.456 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.456 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:14.456 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.150733+0000", 2024-04-19T02:04:14.456 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5960000000000004e-06 2024-04-19T02:04:14.456 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.456 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.456 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.456 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.174940+0000", 2024-04-19T02:04:14.456 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.024206495000000001 2024-04-19T02:04:14.456 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.456 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.457 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for blocked object", 2024-04-19T02:04:14.457 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.175093+0000", 2024-04-19T02:04:14.457 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015350499999999999 2024-04-19T02:04:14.457 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.457 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.457 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.457 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.222569+0000", 2024-04-19T02:04:14.457 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.047475555000000003 2024-04-19T02:04:14.457 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.457 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.457 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-04-19T02:04:14.457 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.222834+0000", 2024-04-19T02:04:14.457 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00026480600000000003 2024-04-19T02:04:14.457 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.457 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.458 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:14.458 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:00.223044+0000", 2024-04-19T02:04:14.458 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021018999999999999 2024-04-19T02:04:14.458 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.458 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:14.458 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.458 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.458 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.458 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.ds1 80/26 ECSubWrite(tid=268, reqid=osd.8.0:268, at_version=80'108, trim_to=0'0, roll_forward_to=79'104))", 2024-04-19T02:04:14.458 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:04:04.163048+0000", 2024-04-19T02:04:14.458 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 10.208057821000001, 2024-04-19T02:04:14.458 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.030775956, 2024-04-19T02:04:14.458 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:14.458 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:14.459 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:04:14.459 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:14.459 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.459 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:14.459 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.163048+0000", 2024-04-19T02:04:14.459 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.459 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.459 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.459 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:14.459 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.163048+0000", 2024-04-19T02:04:14.459 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.459 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.459 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.459 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:14.459 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.163049+0000", 2024-04-19T02:04:14.460 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.564e-06 2024-04-19T02:04:14.460 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.460 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.460 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:14.460 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.163089+0000", 2024-04-19T02:04:14.460 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9245000000000001e-05 2024-04-19T02:04:14.460 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.460 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.460 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:14.460 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.163131+0000", 2024-04-19T02:04:14.460 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2106e-05 2024-04-19T02:04:14.460 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.460 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.460 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:14.460 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.163162+0000", 2024-04-19T02:04:14.461 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1599e-05 2024-04-19T02:04:14.461 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.461 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.461 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.461 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.169994+0000", 2024-04-19T02:04:14.461 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0068321270000000003 2024-04-19T02:04:14.461 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.461 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.461 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:14.461 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.170022+0000", 2024-04-19T02:04:14.461 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8048999999999999e-05 2024-04-19T02:04:14.461 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.461 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.461 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:14.461 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.193700+0000", 2024-04-19T02:04:14.461 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.023677797 2024-04-19T02:04:14.462 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.462 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.462 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:14.462 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.193824+0000", 2024-04-19T02:04:14.462 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012346899999999999 2024-04-19T02:04:14.462 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.462 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:14.462 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.462 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.462 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.462 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s6 80/26 ECSubWrite(tid=350, reqid=osd.3.0:350, at_version=80'184, trim_to=0'0, roll_forward_to=79'182))", 2024-04-19T02:04:14.462 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:04:04.163622+0000", 2024-04-19T02:04:14.462 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 10.207483784000001, 2024-04-19T02:04:14.462 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.038135270999999998, 2024-04-19T02:04:14.463 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:14.463 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:14.463 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:04:14.463 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:14.463 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.463 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:14.463 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.163622+0000", 2024-04-19T02:04:14.463 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.463 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.463 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.463 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:14.463 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.163622+0000", 2024-04-19T02:04:14.463 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.463 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.463 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.464 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:14.464 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.163622+0000", 2024-04-19T02:04:14.464 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4000000000000001e-07 2024-04-19T02:04:14.464 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.464 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.464 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:14.464 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.163681+0000", 2024-04-19T02:04:14.464 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8525e-05 2024-04-19T02:04:14.464 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.464 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.464 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:14.464 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.163712+0000", 2024-04-19T02:04:14.464 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1232999999999997e-05 2024-04-19T02:04:14.464 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.464 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.465 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:14.465 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.163736+0000", 2024-04-19T02:04:14.465 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3305000000000001e-05 2024-04-19T02:04:14.465 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.465 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.465 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.465 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.163992+0000", 2024-04-19T02:04:14.465 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00025603699999999997 2024-04-19T02:04:14.465 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.465 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.465 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:14.465 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.164020+0000", 2024-04-19T02:04:14.465 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8204999999999998e-05 2024-04-19T02:04:14.465 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.465 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.465 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:14.466 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.201674+0000", 2024-04-19T02:04:14.466 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.037654677999999997 2024-04-19T02:04:14.466 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.466 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.466 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:14.466 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.201757+0000", 2024-04-19T02:04:14.466 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2547999999999996e-05 2024-04-19T02:04:14.466 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.466 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:14.466 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.466 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.466 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.466 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDECSubOpWrite(3.2s6 80/26 ECSubWrite(tid=352, reqid=osd.3.0:352, at_version=80'186, trim_to=0'0, roll_forward_to=79'182))", 2024-04-19T02:04:14.467 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-04-19T02:04:04.163794+0000", 2024-04-19T02:04:14.467 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 10.207311702, 2024-04-19T02:04:14.467 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.038018095000000002, 2024-04-19T02:04:14.467 INFO:teuthology.orchestra.run.smithi179.stdout: "continuous": false, 2024-04-19T02:04:14.467 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-04-19T02:04:14.467 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-04-19T02:04:14.467 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-04-19T02:04:14.467 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.467 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-04-19T02:04:14.467 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.163794+0000", 2024-04-19T02:04:14.467 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.467 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.469 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.469 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-04-19T02:04:14.469 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.163794+0000", 2024-04-19T02:04:14.469 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-04-19T02:04:14.469 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.470 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.470 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-04-19T02:04:14.470 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.163795+0000", 2024-04-19T02:04:14.470 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.153e-06 2024-04-19T02:04:14.470 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.470 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.470 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-04-19T02:04:14.470 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.163836+0000", 2024-04-19T02:04:14.470 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1248999999999998e-05 2024-04-19T02:04:14.470 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.470 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.470 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-04-19T02:04:14.470 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.163860+0000", 2024-04-19T02:04:14.470 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3359000000000001e-05 2024-04-19T02:04:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-04-19T02:04:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.163885+0000", 2024-04-19T02:04:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5415999999999999e-05 2024-04-19T02:04:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-04-19T02:04:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.180261+0000", 2024-04-19T02:04:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016376363000000001 2024-04-19T02:04:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_started", 2024-04-19T02:04:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.180289+0000", 2024-04-19T02:04:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7492999999999999e-05 2024-04-19T02:04:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.472 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.472 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_committed", 2024-04-19T02:04:14.472 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.201765+0000", 2024-04-19T02:04:14.472 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021476498 2024-04-19T02:04:14.472 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-04-19T02:04:14.472 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-04-19T02:04:14.472 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-04-19T02:04:14.472 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-04-19T02:04:04.201812+0000", 2024-04-19T02:04:14.472 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6563999999999997e-05 2024-04-19T02:04:14.472 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.472 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:14.472 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.472 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-04-19T02:04:14.472 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-19T02:04:14.472 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-19T02:04:15.102 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is unset 2024-04-19T02:04:15.461 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-04-19T02:04:15.612 INFO:teuthology.orchestra.run.smithi059.stdout:mon.a: {} 2024-04-19T02:04:15.613 INFO:teuthology.orchestra.run.smithi059.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-19T02:04:15.626 INFO:teuthology.orchestra.run.smithi059.stdout:mon.b: {} 2024-04-19T02:04:15.626 INFO:teuthology.orchestra.run.smithi059.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-19T02:04:15.640 INFO:teuthology.orchestra.run.smithi059.stdout:mon.c: {} 2024-04-19T02:04:15.641 INFO:teuthology.orchestra.run.smithi059.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-19T02:04:15.662 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-19T02:04:16.110 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.0 to 1 (10000) 2024-04-19T02:04:16.129 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-04-19T02:04:17.118 INFO:teuthology.orchestra.run.smithi059.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-04-19T02:04:17.129 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-19T02:04:17.143 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-19T02:04:18.122 INFO:teuthology.orchestra.run.smithi059.stderr:noscrub is unset 2024-04-19T02:04:18.122 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.1 to 1 (10000) 2024-04-19T02:04:18.151 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-19T02:04:18.152 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-04-19T02:04:19.138 INFO:teuthology.orchestra.run.smithi059.stderr:nodeep-scrub is unset 2024-04-19T02:04:19.138 INFO:teuthology.orchestra.run.smithi059.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-04-19T02:04:19.170 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-19T02:04:20.141 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.2 to 1 (10000) 2024-04-19T02:04:20.166 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-04-19T02:04:21.157 INFO:teuthology.orchestra.run.smithi059.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-04-19T02:04:21.182 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-19T02:04:22.160 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.3 to 1 (10000) 2024-04-19T02:04:22.188 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-04-19T02:04:23.174 INFO:teuthology.orchestra.run.smithi059.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-04-19T02:04:23.200 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-19T02:04:24.176 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.4 to 1 (10000) 2024-04-19T02:04:24.200 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-04-19T02:04:25.194 INFO:teuthology.orchestra.run.smithi059.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-04-19T02:04:25.220 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-04-19T02:04:26.197 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.5 to 1 (10000) 2024-04-19T02:04:26.221 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-04-19T02:04:27.216 INFO:teuthology.orchestra.run.smithi059.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-04-19T02:04:27.240 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-04-19T02:04:28.218 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.6 to 1 (10000) 2024-04-19T02:04:28.246 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-04-19T02:04:29.234 INFO:teuthology.orchestra.run.smithi059.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-04-19T02:04:29.261 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-04-19T02:04:30.244 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.7 to 1 (10000) 2024-04-19T02:04:30.270 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-04-19T02:04:31.253 INFO:teuthology.orchestra.run.smithi059.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-04-19T02:04:31.278 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-04-19T02:04:32.261 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.8 to 1 (10000) 2024-04-19T02:04:32.290 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 8 1 2024-04-19T02:04:33.263 INFO:teuthology.orchestra.run.smithi059.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-04-19T02:04:33.288 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-04-19T02:04:34.273 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.9 to 1 (10000) 2024-04-19T02:04:34.298 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 9 1 2024-04-19T02:04:35.287 INFO:teuthology.orchestra.run.smithi059.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-04-19T02:04:35.316 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-04-19T02:04:36.290 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.10 to 1 (10000) 2024-04-19T02:04:36.313 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 10 1 2024-04-19T02:04:37.300 INFO:teuthology.orchestra.run.smithi059.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-04-19T02:04:37.328 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-04-19T02:04:38.309 INFO:teuthology.orchestra.run.smithi059.stderr:reweighted osd.11 to 1 (10000) 2024-04-19T02:04:38.334 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 11 1 2024-04-19T02:04:39.326 INFO:teuthology.orchestra.run.smithi059.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-04-19T02:04:39.349 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-04-19T02:04:39.349 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-04-19T02:04:39.349 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-04-19T02:04:39.349 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-04-19T02:04:39.350 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-19T02:04:39.350 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-19T02:04:39.657 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:04:39.657 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":115,"fsid":"db9b8c36-b038-40a6-b749-a60ca7443e45","created":"2024-04-19T02:02:47.352143+0000","modified":"2024-04-19T02:04:39.321038+0000","last_up_change":"2024-04-19T02:02:59.513773+0000","last_in_change":"2024-04-19T02:02:51.039271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-19T02:02:58.287078+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-19T02:03:00.613285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.5199999809265137,"score_stable":4.5199999809265137,"optimal_score":0.82999998331069946,"raw_score_acting":3.75,"raw_score_stable":3.75,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dff2e9d5-c79f-4cc8-b68c-d1d35cc3dfeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1145933874},{"type":"v1","addr":"172.21.15.59:6809","nonce":1145933874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1145933874},{"type":"v1","addr":"172.21.15.59:6811","nonce":1145933874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1145933874},{"type":"v1","addr":"172.21.15.59:6815","nonce":1145933874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1145933874},{"type":"v1","addr":"172.21.15.59:6813","nonce":1145933874}]},"public_addr":"172.21.15.59:6809/1145933874","cluster_addr":"172.21.15.59:6811/1145933874","heartbeat_back_addr":"172.21.15.59:6815/1145933874","heartbeat_front_addr":"172.21.15.59:6813/1145933874","state":["exists","up"]},{"osd":1,"uuid":"30cb7aa1-0bf8-4bc8-8bc5-f8b59870624e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":3807848039},{"type":"v1","addr":"172.21.15.59:6801","nonce":3807848039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":3807848039},{"type":"v1","addr":"172.21.15.59:6803","nonce":3807848039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":3807848039},{"type":"v1","addr":"172.21.15.59:6807","nonce":3807848039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":3807848039},{"type":"v1","addr":"172.21.15.59:6805","nonce":3807848039}]},"public_addr":"172.21.15.59:6801/3807848039","cluster_addr":"172.21.15.59:6803/3807848039","heartbeat_back_addr":"172.21.15.59:6807/3807848039","heartbeat_front_addr":"172.21.15.59:6805/3807848039","state":["exists","up"]},{"osd":2,"uuid":"2e8fca1b-e008-4e58-865e-09a658f6edd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":420007638},{"type":"v1","addr":"172.21.15.59:6817","nonce":420007638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":420007638},{"type":"v1","addr":"172.21.15.59:6822","nonce":420007638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6829","nonce":420007638},{"type":"v1","addr":"172.21.15.59:6831","nonce":420007638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":420007638},{"type":"v1","addr":"172.21.15.59:6827","nonce":420007638}]},"public_addr":"172.21.15.59:6817/420007638","cluster_addr":"172.21.15.59:6822/420007638","heartbeat_back_addr":"172.21.15.59:6831/420007638","heartbeat_front_addr":"172.21.15.59:6827/420007638","state":["exists","up"]},{"osd":3,"uuid":"32289c9a-b7d9-4bcd-9f66-fa68f0c9796b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1791539182},{"type":"v1","addr":"172.21.15.59:6819","nonce":1791539182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6821","nonce":1791539182},{"type":"v1","addr":"172.21.15.59:6823","nonce":1791539182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1791539182},{"type":"v1","addr":"172.21.15.59:6830","nonce":1791539182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6825","nonce":1791539182},{"type":"v1","addr":"172.21.15.59:6826","nonce":1791539182}]},"public_addr":"172.21.15.59:6819/1791539182","cluster_addr":"172.21.15.59:6823/1791539182","heartbeat_back_addr":"172.21.15.59:6830/1791539182","heartbeat_front_addr":"172.21.15.59:6826/1791539182","state":["exists","up"]},{"osd":4,"uuid":"844daeea-860b-4853-a759-994531e5de3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":2921507000},{"type":"v1","addr":"172.21.15.81:6808","nonce":2921507000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":2921507000},{"type":"v1","addr":"172.21.15.81:6811","nonce":2921507000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":2921507000},{"type":"v1","addr":"172.21.15.81:6815","nonce":2921507000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2921507000},{"type":"v1","addr":"172.21.15.81:6813","nonce":2921507000}]},"public_addr":"172.21.15.81:6808/2921507000","cluster_addr":"172.21.15.81:6811/2921507000","heartbeat_back_addr":"172.21.15.81:6815/2921507000","heartbeat_front_addr":"172.21.15.81:6813/2921507000","state":["exists","up"]},{"osd":5,"uuid":"f43ad412-74bb-43a6-95e7-25ee06bde025","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3925467179},{"type":"v1","addr":"172.21.15.81:6818","nonce":3925467179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":3925467179},{"type":"v1","addr":"172.21.15.81:6821","nonce":3925467179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":3925467179},{"type":"v1","addr":"172.21.15.81:6826","nonce":3925467179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":3925467179},{"type":"v1","addr":"172.21.15.81:6824","nonce":3925467179}]},"public_addr":"172.21.15.81:6818/3925467179","cluster_addr":"172.21.15.81:6821/3925467179","heartbeat_back_addr":"172.21.15.81:6826/3925467179","heartbeat_front_addr":"172.21.15.81:6824/3925467179","state":["exists","up"]},{"osd":6,"uuid":"07ccbc36-1f5d-4595-8578-09362b5e212f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3714790214},{"type":"v1","addr":"172.21.15.81:6801","nonce":3714790214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3714790214},{"type":"v1","addr":"172.21.15.81:6803","nonce":3714790214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6807","nonce":3714790214},{"type":"v1","addr":"172.21.15.81:6809","nonce":3714790214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3714790214},{"type":"v1","addr":"172.21.15.81:6805","nonce":3714790214}]},"public_addr":"172.21.15.81:6801/3714790214","cluster_addr":"172.21.15.81:6803/3714790214","heartbeat_back_addr":"172.21.15.81:6809/3714790214","heartbeat_front_addr":"172.21.15.81:6805/3714790214","state":["exists","up"]},{"osd":7,"uuid":"6b2247e7-ebf8-4c0e-89ae-e15985f59aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":2170932617},{"type":"v1","addr":"172.21.15.81:6820","nonce":2170932617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":2170932617},{"type":"v1","addr":"172.21.15.81:6827","nonce":2170932617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":2170932617},{"type":"v1","addr":"172.21.15.81:6831","nonce":2170932617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":2170932617},{"type":"v1","addr":"172.21.15.81:6829","nonce":2170932617}]},"public_addr":"172.21.15.81:6820/2170932617","cluster_addr":"172.21.15.81:6827/2170932617","heartbeat_back_addr":"172.21.15.81:6831/2170932617","heartbeat_front_addr":"172.21.15.81:6829/2170932617","state":["exists","up"]},{"osd":8,"uuid":"7142efd0-2144-4973-a01d-69a064b9b51f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2053612068},{"type":"v1","addr":"172.21.15.179:6825","nonce":2053612068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2053612068},{"type":"v1","addr":"172.21.15.179:6827","nonce":2053612068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2053612068},{"type":"v1","addr":"172.21.15.179:6831","nonce":2053612068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2053612068},{"type":"v1","addr":"172.21.15.179:6829","nonce":2053612068}]},"public_addr":"172.21.15.179:6825/2053612068","cluster_addr":"172.21.15.179:6827/2053612068","heartbeat_back_addr":"172.21.15.179:6831/2053612068","heartbeat_front_addr":"172.21.15.179:6829/2053612068","state":["exists","up"]},{"osd":9,"uuid":"30a29baa-53cc-4c29-ab41-af14ead5c539","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2641050728},{"type":"v1","addr":"172.21.15.179:6801","nonce":2641050728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2641050728},{"type":"v1","addr":"172.21.15.179:6803","nonce":2641050728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2641050728},{"type":"v1","addr":"172.21.15.179:6807","nonce":2641050728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2641050728},{"type":"v1","addr":"172.21.15.179:6805","nonce":2641050728}]},"public_addr":"172.21.15.179:6801/2641050728","cluster_addr":"172.21.15.179:6803/2641050728","heartbeat_back_addr":"172.21.15.179:6807/2641050728","heartbeat_front_addr":"172.21.15.179:6805/2641050728","state":["exists","up"]},{"osd":10,"uuid":"fae84343-8b7a-4bd3-aeef-823f2f6058e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3465727116},{"type":"v1","addr":"172.21.15.179:6817","nonce":3465727116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3465727116},{"type":"v1","addr":"172.21.15.179:6819","nonce":3465727116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3465727116},{"type":"v1","addr":"172.21.15.179:6823","nonce":3465727116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3465727116},{"type":"v1","addr":"172.21.15.179:6821","nonce":3465727116}]},"public_addr":"172.21.15.179:6817/3465727116","cluster_addr":"172.21.15.179:6819/3465727116","heartbeat_back_addr":"172.21.15.179:6823/3465727116","heartbeat_front_addr":"172.21.15.179:6821/3465727116","state":["exists","up"]},{"osd":11,"uuid":"35858134-6864-4da1-88c2-4b48166febab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":4047209718},{"type":"v1","addr":"172.21.15.179:6809","nonce":4047209718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":4047209718},{"type":"v1","addr":"172.21.15.179:6811","nonce":4047209718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4047209718},{"type":"v1","addr":"172.21.15.179:6815","nonce":4047209718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4047209718},{"type":"v1","addr":"172.21.15.179:6813","nonce":4047209718}]},"public_addr":"172.21.15.179:6809/4047209718","cluster_addr":"172.21.15.179:6811/4047209718","heartbeat_back_addr":"172.21.15.179:6815/4047209718","heartbeat_front_addr":"172.21.15.179:6813/4047209718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:12.165335+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:26.292747+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:16.779735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:14.023955+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:15.627221+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:15.527302+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:16.257148+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:12.903521+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:17.699941+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:23.768672+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:17.057934+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:03:57.837672+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lrcprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","l":"3","m":"2","name":"lrcprofile","plugin":"lrc"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":27,"length":1},{"begin":29,"length":1},{"begin":33,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T02:04:39.678 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-19T02:04:39.678 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-19T02:04:39.981 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:04:39.981 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":115,"fsid":"db9b8c36-b038-40a6-b749-a60ca7443e45","created":"2024-04-19T02:02:47.352143+0000","modified":"2024-04-19T02:04:39.321038+0000","last_up_change":"2024-04-19T02:02:59.513773+0000","last_in_change":"2024-04-19T02:02:51.039271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-19T02:02:58.287078+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-19T02:03:00.613285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.5199999809265137,"score_stable":4.5199999809265137,"optimal_score":0.82999998331069946,"raw_score_acting":3.75,"raw_score_stable":3.75,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dff2e9d5-c79f-4cc8-b68c-d1d35cc3dfeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1145933874},{"type":"v1","addr":"172.21.15.59:6809","nonce":1145933874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1145933874},{"type":"v1","addr":"172.21.15.59:6811","nonce":1145933874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1145933874},{"type":"v1","addr":"172.21.15.59:6815","nonce":1145933874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1145933874},{"type":"v1","addr":"172.21.15.59:6813","nonce":1145933874}]},"public_addr":"172.21.15.59:6809/1145933874","cluster_addr":"172.21.15.59:6811/1145933874","heartbeat_back_addr":"172.21.15.59:6815/1145933874","heartbeat_front_addr":"172.21.15.59:6813/1145933874","state":["exists","up"]},{"osd":1,"uuid":"30cb7aa1-0bf8-4bc8-8bc5-f8b59870624e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":3807848039},{"type":"v1","addr":"172.21.15.59:6801","nonce":3807848039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":3807848039},{"type":"v1","addr":"172.21.15.59:6803","nonce":3807848039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":3807848039},{"type":"v1","addr":"172.21.15.59:6807","nonce":3807848039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":3807848039},{"type":"v1","addr":"172.21.15.59:6805","nonce":3807848039}]},"public_addr":"172.21.15.59:6801/3807848039","cluster_addr":"172.21.15.59:6803/3807848039","heartbeat_back_addr":"172.21.15.59:6807/3807848039","heartbeat_front_addr":"172.21.15.59:6805/3807848039","state":["exists","up"]},{"osd":2,"uuid":"2e8fca1b-e008-4e58-865e-09a658f6edd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":420007638},{"type":"v1","addr":"172.21.15.59:6817","nonce":420007638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":420007638},{"type":"v1","addr":"172.21.15.59:6822","nonce":420007638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6829","nonce":420007638},{"type":"v1","addr":"172.21.15.59:6831","nonce":420007638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":420007638},{"type":"v1","addr":"172.21.15.59:6827","nonce":420007638}]},"public_addr":"172.21.15.59:6817/420007638","cluster_addr":"172.21.15.59:6822/420007638","heartbeat_back_addr":"172.21.15.59:6831/420007638","heartbeat_front_addr":"172.21.15.59:6827/420007638","state":["exists","up"]},{"osd":3,"uuid":"32289c9a-b7d9-4bcd-9f66-fa68f0c9796b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1791539182},{"type":"v1","addr":"172.21.15.59:6819","nonce":1791539182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6821","nonce":1791539182},{"type":"v1","addr":"172.21.15.59:6823","nonce":1791539182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1791539182},{"type":"v1","addr":"172.21.15.59:6830","nonce":1791539182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6825","nonce":1791539182},{"type":"v1","addr":"172.21.15.59:6826","nonce":1791539182}]},"public_addr":"172.21.15.59:6819/1791539182","cluster_addr":"172.21.15.59:6823/1791539182","heartbeat_back_addr":"172.21.15.59:6830/1791539182","heartbeat_front_addr":"172.21.15.59:6826/1791539182","state":["exists","up"]},{"osd":4,"uuid":"844daeea-860b-4853-a759-994531e5de3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":2921507000},{"type":"v1","addr":"172.21.15.81:6808","nonce":2921507000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":2921507000},{"type":"v1","addr":"172.21.15.81:6811","nonce":2921507000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":2921507000},{"type":"v1","addr":"172.21.15.81:6815","nonce":2921507000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2921507000},{"type":"v1","addr":"172.21.15.81:6813","nonce":2921507000}]},"public_addr":"172.21.15.81:6808/2921507000","cluster_addr":"172.21.15.81:6811/2921507000","heartbeat_back_addr":"172.21.15.81:6815/2921507000","heartbeat_front_addr":"172.21.15.81:6813/2921507000","state":["exists","up"]},{"osd":5,"uuid":"f43ad412-74bb-43a6-95e7-25ee06bde025","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3925467179},{"type":"v1","addr":"172.21.15.81:6818","nonce":3925467179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":3925467179},{"type":"v1","addr":"172.21.15.81:6821","nonce":3925467179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":3925467179},{"type":"v1","addr":"172.21.15.81:6826","nonce":3925467179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":3925467179},{"type":"v1","addr":"172.21.15.81:6824","nonce":3925467179}]},"public_addr":"172.21.15.81:6818/3925467179","cluster_addr":"172.21.15.81:6821/3925467179","heartbeat_back_addr":"172.21.15.81:6826/3925467179","heartbeat_front_addr":"172.21.15.81:6824/3925467179","state":["exists","up"]},{"osd":6,"uuid":"07ccbc36-1f5d-4595-8578-09362b5e212f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3714790214},{"type":"v1","addr":"172.21.15.81:6801","nonce":3714790214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3714790214},{"type":"v1","addr":"172.21.15.81:6803","nonce":3714790214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6807","nonce":3714790214},{"type":"v1","addr":"172.21.15.81:6809","nonce":3714790214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3714790214},{"type":"v1","addr":"172.21.15.81:6805","nonce":3714790214}]},"public_addr":"172.21.15.81:6801/3714790214","cluster_addr":"172.21.15.81:6803/3714790214","heartbeat_back_addr":"172.21.15.81:6809/3714790214","heartbeat_front_addr":"172.21.15.81:6805/3714790214","state":["exists","up"]},{"osd":7,"uuid":"6b2247e7-ebf8-4c0e-89ae-e15985f59aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":2170932617},{"type":"v1","addr":"172.21.15.81:6820","nonce":2170932617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":2170932617},{"type":"v1","addr":"172.21.15.81:6827","nonce":2170932617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":2170932617},{"type":"v1","addr":"172.21.15.81:6831","nonce":2170932617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":2170932617},{"type":"v1","addr":"172.21.15.81:6829","nonce":2170932617}]},"public_addr":"172.21.15.81:6820/2170932617","cluster_addr":"172.21.15.81:6827/2170932617","heartbeat_back_addr":"172.21.15.81:6831/2170932617","heartbeat_front_addr":"172.21.15.81:6829/2170932617","state":["exists","up"]},{"osd":8,"uuid":"7142efd0-2144-4973-a01d-69a064b9b51f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2053612068},{"type":"v1","addr":"172.21.15.179:6825","nonce":2053612068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2053612068},{"type":"v1","addr":"172.21.15.179:6827","nonce":2053612068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2053612068},{"type":"v1","addr":"172.21.15.179:6831","nonce":2053612068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2053612068},{"type":"v1","addr":"172.21.15.179:6829","nonce":2053612068}]},"public_addr":"172.21.15.179:6825/2053612068","cluster_addr":"172.21.15.179:6827/2053612068","heartbeat_back_addr":"172.21.15.179:6831/2053612068","heartbeat_front_addr":"172.21.15.179:6829/2053612068","state":["exists","up"]},{"osd":9,"uuid":"30a29baa-53cc-4c29-ab41-af14ead5c539","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2641050728},{"type":"v1","addr":"172.21.15.179:6801","nonce":2641050728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2641050728},{"type":"v1","addr":"172.21.15.179:6803","nonce":2641050728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2641050728},{"type":"v1","addr":"172.21.15.179:6807","nonce":2641050728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2641050728},{"type":"v1","addr":"172.21.15.179:6805","nonce":2641050728}]},"public_addr":"172.21.15.179:6801/2641050728","cluster_addr":"172.21.15.179:6803/2641050728","heartbeat_back_addr":"172.21.15.179:6807/2641050728","heartbeat_front_addr":"172.21.15.179:6805/2641050728","state":["exists","up"]},{"osd":10,"uuid":"fae84343-8b7a-4bd3-aeef-823f2f6058e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3465727116},{"type":"v1","addr":"172.21.15.179:6817","nonce":3465727116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3465727116},{"type":"v1","addr":"172.21.15.179:6819","nonce":3465727116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3465727116},{"type":"v1","addr":"172.21.15.179:6823","nonce":3465727116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3465727116},{"type":"v1","addr":"172.21.15.179:6821","nonce":3465727116}]},"public_addr":"172.21.15.179:6817/3465727116","cluster_addr":"172.21.15.179:6819/3465727116","heartbeat_back_addr":"172.21.15.179:6823/3465727116","heartbeat_front_addr":"172.21.15.179:6821/3465727116","state":["exists","up"]},{"osd":11,"uuid":"35858134-6864-4da1-88c2-4b48166febab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":4047209718},{"type":"v1","addr":"172.21.15.179:6809","nonce":4047209718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":4047209718},{"type":"v1","addr":"172.21.15.179:6811","nonce":4047209718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4047209718},{"type":"v1","addr":"172.21.15.179:6815","nonce":4047209718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4047209718},{"type":"v1","addr":"172.21.15.179:6813","nonce":4047209718}]},"public_addr":"172.21.15.179:6809/4047209718","cluster_addr":"172.21.15.179:6811/4047209718","heartbeat_back_addr":"172.21.15.179:6815/4047209718","heartbeat_front_addr":"172.21.15.179:6813/4047209718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:12.165335+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:26.292747+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:16.779735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:14.023955+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:15.627221+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:15.527302+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:16.257148+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:12.903521+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:17.699941+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:23.768672+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:17.057934+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:03:57.837672+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lrcprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","l":"3","m":"2","name":"lrcprofile","plugin":"lrc"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":27,"length":1},{"begin":29,"length":1},{"begin":33,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T02:04:40.002 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-19T02:04:40.002 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-19T02:04:40.003 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-19T02:04:40.003 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-19T02:04:40.004 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-19T02:04:40.004 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-19T02:04:40.004 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-19T02:04:40.005 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-19T02:04:40.005 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-04-19T02:04:40.006 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-04-19T02:04:40.006 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-04-19T02:04:40.006 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-04-19T02:04:40.351 INFO:teuthology.orchestra.run.smithi059.stdout:73014444056 2024-04-19T02:04:40.352 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-19T02:04:40.360 INFO:teuthology.orchestra.run.smithi059.stdout:81604378647 2024-04-19T02:04:40.360 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-19T02:04:40.371 INFO:teuthology.orchestra.run.smithi059.stdout:77309411352 2024-04-19T02:04:40.372 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-19T02:04:40.373 INFO:teuthology.orchestra.run.smithi059.stdout:77309411352 2024-04-19T02:04:40.374 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-04-19T02:04:40.383 INFO:teuthology.orchestra.run.smithi059.stdout:77309411351 2024-04-19T02:04:40.383 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-19T02:04:40.389 INFO:teuthology.orchestra.run.smithi059.stdout:73014444056 2024-04-19T02:04:40.390 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-19T02:04:40.433 INFO:teuthology.orchestra.run.smithi059.stdout:77309411353 2024-04-19T02:04:40.434 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-04-19T02:04:40.436 INFO:teuthology.orchestra.run.smithi059.stdout:81604378648 2024-04-19T02:04:40.436 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-19T02:04:40.444 INFO:teuthology.orchestra.run.smithi059.stdout:77309411353 2024-04-19T02:04:40.445 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-19T02:04:40.454 INFO:teuthology.orchestra.run.smithi059.stdout:77309411352 2024-04-19T02:04:40.455 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-19T02:04:40.471 INFO:teuthology.orchestra.run.smithi059.stdout:77309411353 2024-04-19T02:04:40.471 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-04-19T02:04:40.474 INFO:teuthology.orchestra.run.smithi059.stdout:73014444056 2024-04-19T02:04:40.474 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-04-19T02:04:41.015 INFO:teuthology.orchestra.run.smithi059.stdout:73014444055 2024-04-19T02:04:41.059 INFO:teuthology.orchestra.run.smithi059.stdout:73014444055 2024-04-19T02:04:41.079 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444056 got 73014444055 for osd.5 2024-04-19T02:04:41.099 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444056 got 73014444055 for osd.1 2024-04-19T02:04:41.135 INFO:teuthology.orchestra.run.smithi059.stdout:73014444055 2024-04-19T02:04:41.151 INFO:teuthology.orchestra.run.smithi059.stdout:77309411351 2024-04-19T02:04:41.152 INFO:teuthology.orchestra.run.smithi059.stdout:81604378646 2024-04-19T02:04:41.154 INFO:teuthology.orchestra.run.smithi059.stdout:77309411351 2024-04-19T02:04:41.173 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444056 got 73014444055 for osd.10 2024-04-19T02:04:41.187 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411352 got 77309411351 for osd.3 2024-04-19T02:04:41.188 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411352 got 77309411351 for osd.11 2024-04-19T02:04:41.198 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378647 got 81604378646 for osd.0 2024-04-19T02:04:41.199 INFO:teuthology.orchestra.run.smithi059.stdout:77309411350 2024-04-19T02:04:41.201 INFO:teuthology.orchestra.run.smithi059.stdout:77309411352 2024-04-19T02:04:41.228 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411351 got 77309411350 for osd.6 2024-04-19T02:04:41.229 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411353 got 77309411352 for osd.8 2024-04-19T02:04:41.241 INFO:teuthology.orchestra.run.smithi059.stdout:81604378647 2024-04-19T02:04:41.256 INFO:teuthology.orchestra.run.smithi059.stdout:77309411352 2024-04-19T02:04:41.260 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378648 got 81604378647 for osd.2 2024-04-19T02:04:41.279 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411353 got 77309411352 for osd.7 2024-04-19T02:04:41.289 INFO:teuthology.orchestra.run.smithi059.stdout:77309411352 2024-04-19T02:04:41.294 INFO:teuthology.orchestra.run.smithi059.stdout:77309411351 2024-04-19T02:04:41.308 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411353 got 77309411352 for osd.9 2024-04-19T02:04:41.313 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411352 got 77309411351 for osd.4 2024-04-19T02:04:42.080 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-19T02:04:42.099 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-19T02:04:42.174 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-04-19T02:04:42.187 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-19T02:04:42.190 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-04-19T02:04:42.199 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-19T02:04:42.228 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-19T02:04:42.229 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-04-19T02:04:42.262 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-19T02:04:42.281 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-19T02:04:42.308 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-04-19T02:04:42.314 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-19T02:04:42.592 INFO:teuthology.orchestra.run.smithi059.stdout:73014444056 2024-04-19T02:04:42.625 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444056 got 73014444056 for osd.5 2024-04-19T02:04:42.625 DEBUG:teuthology.parallel:result is None 2024-04-19T02:04:42.754 INFO:teuthology.orchestra.run.smithi059.stdout:73014444056 2024-04-19T02:04:42.786 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444056 got 73014444056 for osd.1 2024-04-19T02:04:42.786 DEBUG:teuthology.parallel:result is None 2024-04-19T02:04:42.810 INFO:teuthology.orchestra.run.smithi059.stdout:73014444056 2024-04-19T02:04:42.865 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444056 got 73014444056 for osd.10 2024-04-19T02:04:42.865 INFO:teuthology.orchestra.run.smithi059.stdout:77309411352 2024-04-19T02:04:42.865 DEBUG:teuthology.parallel:result is None 2024-04-19T02:04:42.875 INFO:teuthology.orchestra.run.smithi059.stdout:77309411352 2024-04-19T02:04:42.899 INFO:teuthology.orchestra.run.smithi059.stdout:81604378648 2024-04-19T02:04:42.903 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411352 got 77309411352 for osd.3 2024-04-19T02:04:42.903 DEBUG:teuthology.parallel:result is None 2024-04-19T02:04:42.920 INFO:teuthology.orchestra.run.smithi059.stdout:77309411351 2024-04-19T02:04:42.926 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411352 got 77309411352 for osd.11 2024-04-19T02:04:42.926 DEBUG:teuthology.parallel:result is None 2024-04-19T02:04:42.928 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378647 got 81604378648 for osd.0 2024-04-19T02:04:42.928 DEBUG:teuthology.parallel:result is None 2024-04-19T02:04:42.932 INFO:teuthology.orchestra.run.smithi059.stdout:81604378649 2024-04-19T02:04:42.950 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411351 got 77309411351 for osd.6 2024-04-19T02:04:42.950 DEBUG:teuthology.parallel:result is None 2024-04-19T02:04:42.958 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378648 got 81604378649 for osd.2 2024-04-19T02:04:42.958 DEBUG:teuthology.parallel:result is None 2024-04-19T02:04:42.981 INFO:teuthology.orchestra.run.smithi059.stdout:77309411353 2024-04-19T02:04:42.998 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411353 got 77309411353 for osd.8 2024-04-19T02:04:42.998 DEBUG:teuthology.parallel:result is None 2024-04-19T02:04:43.019 INFO:teuthology.orchestra.run.smithi059.stdout:77309411353 2024-04-19T02:04:43.036 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411353 got 77309411353 for osd.7 2024-04-19T02:04:43.036 DEBUG:teuthology.parallel:result is None 2024-04-19T02:04:43.053 INFO:teuthology.orchestra.run.smithi059.stdout:77309411353 2024-04-19T02:04:43.066 INFO:teuthology.orchestra.run.smithi059.stdout:77309411352 2024-04-19T02:04:43.070 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411353 got 77309411353 for osd.9 2024-04-19T02:04:43.070 DEBUG:teuthology.parallel:result is None 2024-04-19T02:04:43.083 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411352 got 77309411352 for osd.4 2024-04-19T02:04:43.083 DEBUG:teuthology.parallel:result is None 2024-04-19T02:04:43.085 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-19T02:04:43.085 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T02:04:43.402 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:04:43.403 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-19T02:04:43.419 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-04-19T02:04:42.233031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":513280,"kb_used_data":16892,"kb_used_omap":18,"kb_used_meta":496173,"kb_avail":1131948800,"statfs":{"total":1159641169920,"available":1159115571200,"internally_reserved":0,"allocated":17297408,"data_stored":11158916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":508081465},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.939715"},"pg_stats":[{"pgid":"1.0","version":"96'139","reported_seq":384,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.333126+0000","last_change":"2024-04-19T02:04:20.382306+0000","last_active":"2024-04-19T02:04:39.333126+0000","last_peered":"2024-04-19T02:04:39.333126+0000","last_clean":"2024-04-19T02:04:39.333126+0000","last_became_active":"2024-04-19T02:02:59.972005+0000","last_became_peered":"2024-04-19T02:02:59.972005+0000","last_unstale":"2024-04-19T02:04:39.333126+0000","last_undegraded":"2024-04-19T02:04:39.333126+0000","last_fullsized":"2024-04-19T02:04:39.333126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'139","last_scrub_stamp":"2024-04-19T02:04:20.382039+0000","last_deep_scrub":"96'139","last_deep_scrub_stamp":"2024-04-19T02:04:20.382039+0000","last_clean_scrub_stamp":"2024-04-19T02:04:20.382039+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:42.019594+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":201,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.723615+0000","last_change":"2024-04-19T02:04:22.026870+0000","last_active":"2024-04-19T02:04:39.723615+0000","last_peered":"2024-04-19T02:04:39.723615+0000","last_clean":"2024-04-19T02:04:39.723615+0000","last_became_active":"2024-04-19T02:03:03.026063+0000","last_became_peered":"2024-04-19T02:03:03.026063+0000","last_unstale":"2024-04-19T02:04:39.723615+0000","last_undegraded":"2024-04-19T02:04:39.723615+0000","last_fullsized":"2024-04-19T02:04:39.723615+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:22.026513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:22.026513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:46.328574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":203,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.332912+0000","last_change":"2024-04-19T02:04:17.332692+0000","last_active":"2024-04-19T02:04:39.332912+0000","last_peered":"2024-04-19T02:04:39.332912+0000","last_clean":"2024-04-19T02:04:39.332912+0000","last_became_active":"2024-04-19T02:03:02.560210+0000","last_became_peered":"2024-04-19T02:03:02.560210+0000","last_unstale":"2024-04-19T02:04:39.332912+0000","last_undegraded":"2024-04-19T02:04:39.332912+0000","last_fullsized":"2024-04-19T02:04:39.332912+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:17.332474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:17.332474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:44.300211+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":60,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.722031+0000","last_change":"2024-04-19T02:04:26.463553+0000","last_active":"2024-04-19T02:04:39.722031+0000","last_peered":"2024-04-19T02:04:39.722031+0000","last_clean":"2024-04-19T02:04:39.722031+0000","last_became_active":"2024-04-19T02:04:22.996729+0000","last_became_peered":"2024-04-19T02:04:22.996729+0000","last_unstale":"2024-04-19T02:04:39.722031+0000","last_undegraded":"2024-04-19T02:04:39.722031+0000","last_fullsized":"2024-04-19T02:04:39.722031+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:26.463421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:26.463421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:33.227058+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":203,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.332953+0000","last_change":"2024-04-19T02:04:33.441354+0000","last_active":"2024-04-19T02:04:39.332953+0000","last_peered":"2024-04-19T02:04:39.332953+0000","last_clean":"2024-04-19T02:04:39.332953+0000","last_became_active":"2024-04-19T02:03:02.559377+0000","last_became_peered":"2024-04-19T02:03:02.559377+0000","last_unstale":"2024-04-19T02:04:39.332953+0000","last_undegraded":"2024-04-19T02:04:39.332953+0000","last_fullsized":"2024-04-19T02:04:39.332953+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:33.441118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:33.441118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:36.624187+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":201,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.332093+0000","last_change":"2024-04-19T02:04:16.610529+0000","last_active":"2024-04-19T02:04:39.332093+0000","last_peered":"2024-04-19T02:04:39.332093+0000","last_clean":"2024-04-19T02:04:39.332093+0000","last_became_active":"2024-04-19T02:03:02.597118+0000","last_became_peered":"2024-04-19T02:03:02.597118+0000","last_unstale":"2024-04-19T02:04:39.332093+0000","last_undegraded":"2024-04-19T02:04:39.332093+0000","last_fullsized":"2024-04-19T02:04:39.332093+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:16.610198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:16.610198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:35.025165+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":203,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.724014+0000","last_change":"2024-04-19T02:04:04.753289+0000","last_active":"2024-04-19T02:04:39.724014+0000","last_peered":"2024-04-19T02:04:39.724014+0000","last_clean":"2024-04-19T02:04:39.724014+0000","last_became_active":"2024-04-19T02:03:02.595232+0000","last_became_peered":"2024-04-19T02:03:02.595232+0000","last_unstale":"2024-04-19T02:04:39.724014+0000","last_undegraded":"2024-04-19T02:04:39.724014+0000","last_fullsized":"2024-04-19T02:04:39.724014+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:04.752922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:04.752922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:25.635349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":201,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.332074+0000","last_change":"2024-04-19T02:04:29.771190+0000","last_active":"2024-04-19T02:04:39.332074+0000","last_peered":"2024-04-19T02:04:39.332074+0000","last_clean":"2024-04-19T02:04:39.332074+0000","last_became_active":"2024-04-19T02:03:02.988073+0000","last_became_peered":"2024-04-19T02:03:02.988073+0000","last_unstale":"2024-04-19T02:04:39.332074+0000","last_undegraded":"2024-04-19T02:04:39.332074+0000","last_fullsized":"2024-04-19T02:04:39.332074+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:29.770959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:29.770959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:54.167208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":203,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.332982+0000","last_change":"2024-04-19T02:04:25.439628+0000","last_active":"2024-04-19T02:04:39.332982+0000","last_peered":"2024-04-19T02:04:39.332982+0000","last_clean":"2024-04-19T02:04:39.332982+0000","last_became_active":"2024-04-19T02:03:02.570116+0000","last_became_peered":"2024-04-19T02:03:02.570116+0000","last_unstale":"2024-04-19T02:04:39.332982+0000","last_undegraded":"2024-04-19T02:04:39.332982+0000","last_fullsized":"2024-04-19T02:04:39.332982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:25.439289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:25.439289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:28.101891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":19,"seq":81604378649,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32492,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339348,"statfs":{"total":96636764160,"available":96603492352,"internally_reserved":0,"allocated":1343488,"data_stored":835980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":19,"seq":81604378648,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411352,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50552,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321288,"statfs":{"total":96636764160,"available":96584998912,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411353,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50552,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321288,"statfs":{"total":96636764160,"available":96584998912,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411353,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51076,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320764,"statfs":{"total":96636764160,"available":96584462336,"internally_reserved":0,"allocated":1888256,"data_stored":1372608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411353,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50556,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321284,"statfs":{"total":96636764160,"available":96584994816,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":18,"seq":77309411351,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28332,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343508,"statfs":{"total":96636764160,"available":96607752192,"internally_reserved":0,"allocated":1343488,"data_stored":835980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411352,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32508,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":18,"seq":77309411352,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50556,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321284,"statfs":{"total":96636764160,"available":96584994816,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":17,"seq":73014444056,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32512,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":17,"seq":73014444056,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51072,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320768,"statfs":{"total":96636764160,"available":96584466432,"internally_reserved":0,"allocated":1875968,"data_stored":1364908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":17,"seq":73014444056,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50560,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321280,"statfs":{"total":96636764160,"available":96584990720,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T02:04:43.420 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T02:04:43.699 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:04:43.700 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-19T02:04:43.716 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-04-19T02:04:42.233031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":513280,"kb_used_data":16892,"kb_used_omap":18,"kb_used_meta":496173,"kb_avail":1131948800,"statfs":{"total":1159641169920,"available":1159115571200,"internally_reserved":0,"allocated":17297408,"data_stored":11158916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":508081465},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.939715"},"pg_stats":[{"pgid":"1.0","version":"96'139","reported_seq":384,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.333126+0000","last_change":"2024-04-19T02:04:20.382306+0000","last_active":"2024-04-19T02:04:39.333126+0000","last_peered":"2024-04-19T02:04:39.333126+0000","last_clean":"2024-04-19T02:04:39.333126+0000","last_became_active":"2024-04-19T02:02:59.972005+0000","last_became_peered":"2024-04-19T02:02:59.972005+0000","last_unstale":"2024-04-19T02:04:39.333126+0000","last_undegraded":"2024-04-19T02:04:39.333126+0000","last_fullsized":"2024-04-19T02:04:39.333126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'139","last_scrub_stamp":"2024-04-19T02:04:20.382039+0000","last_deep_scrub":"96'139","last_deep_scrub_stamp":"2024-04-19T02:04:20.382039+0000","last_clean_scrub_stamp":"2024-04-19T02:04:20.382039+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:42.019594+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":201,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.723615+0000","last_change":"2024-04-19T02:04:22.026870+0000","last_active":"2024-04-19T02:04:39.723615+0000","last_peered":"2024-04-19T02:04:39.723615+0000","last_clean":"2024-04-19T02:04:39.723615+0000","last_became_active":"2024-04-19T02:03:03.026063+0000","last_became_peered":"2024-04-19T02:03:03.026063+0000","last_unstale":"2024-04-19T02:04:39.723615+0000","last_undegraded":"2024-04-19T02:04:39.723615+0000","last_fullsized":"2024-04-19T02:04:39.723615+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:22.026513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:22.026513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:46.328574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":203,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.332912+0000","last_change":"2024-04-19T02:04:17.332692+0000","last_active":"2024-04-19T02:04:39.332912+0000","last_peered":"2024-04-19T02:04:39.332912+0000","last_clean":"2024-04-19T02:04:39.332912+0000","last_became_active":"2024-04-19T02:03:02.560210+0000","last_became_peered":"2024-04-19T02:03:02.560210+0000","last_unstale":"2024-04-19T02:04:39.332912+0000","last_undegraded":"2024-04-19T02:04:39.332912+0000","last_fullsized":"2024-04-19T02:04:39.332912+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:17.332474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:17.332474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:44.300211+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":60,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.722031+0000","last_change":"2024-04-19T02:04:26.463553+0000","last_active":"2024-04-19T02:04:39.722031+0000","last_peered":"2024-04-19T02:04:39.722031+0000","last_clean":"2024-04-19T02:04:39.722031+0000","last_became_active":"2024-04-19T02:04:22.996729+0000","last_became_peered":"2024-04-19T02:04:22.996729+0000","last_unstale":"2024-04-19T02:04:39.722031+0000","last_undegraded":"2024-04-19T02:04:39.722031+0000","last_fullsized":"2024-04-19T02:04:39.722031+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:26.463421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:26.463421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:33.227058+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":203,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.332953+0000","last_change":"2024-04-19T02:04:33.441354+0000","last_active":"2024-04-19T02:04:39.332953+0000","last_peered":"2024-04-19T02:04:39.332953+0000","last_clean":"2024-04-19T02:04:39.332953+0000","last_became_active":"2024-04-19T02:03:02.559377+0000","last_became_peered":"2024-04-19T02:03:02.559377+0000","last_unstale":"2024-04-19T02:04:39.332953+0000","last_undegraded":"2024-04-19T02:04:39.332953+0000","last_fullsized":"2024-04-19T02:04:39.332953+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:33.441118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:33.441118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:36.624187+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":201,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.332093+0000","last_change":"2024-04-19T02:04:16.610529+0000","last_active":"2024-04-19T02:04:39.332093+0000","last_peered":"2024-04-19T02:04:39.332093+0000","last_clean":"2024-04-19T02:04:39.332093+0000","last_became_active":"2024-04-19T02:03:02.597118+0000","last_became_peered":"2024-04-19T02:03:02.597118+0000","last_unstale":"2024-04-19T02:04:39.332093+0000","last_undegraded":"2024-04-19T02:04:39.332093+0000","last_fullsized":"2024-04-19T02:04:39.332093+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:16.610198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:16.610198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:35.025165+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":203,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.724014+0000","last_change":"2024-04-19T02:04:04.753289+0000","last_active":"2024-04-19T02:04:39.724014+0000","last_peered":"2024-04-19T02:04:39.724014+0000","last_clean":"2024-04-19T02:04:39.724014+0000","last_became_active":"2024-04-19T02:03:02.595232+0000","last_became_peered":"2024-04-19T02:03:02.595232+0000","last_unstale":"2024-04-19T02:04:39.724014+0000","last_undegraded":"2024-04-19T02:04:39.724014+0000","last_fullsized":"2024-04-19T02:04:39.724014+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:04.752922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:04.752922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:25.635349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":201,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.332074+0000","last_change":"2024-04-19T02:04:29.771190+0000","last_active":"2024-04-19T02:04:39.332074+0000","last_peered":"2024-04-19T02:04:39.332074+0000","last_clean":"2024-04-19T02:04:39.332074+0000","last_became_active":"2024-04-19T02:03:02.988073+0000","last_became_peered":"2024-04-19T02:03:02.988073+0000","last_unstale":"2024-04-19T02:04:39.332074+0000","last_undegraded":"2024-04-19T02:04:39.332074+0000","last_fullsized":"2024-04-19T02:04:39.332074+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:29.770959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:29.770959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:54.167208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":203,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.332982+0000","last_change":"2024-04-19T02:04:25.439628+0000","last_active":"2024-04-19T02:04:39.332982+0000","last_peered":"2024-04-19T02:04:39.332982+0000","last_clean":"2024-04-19T02:04:39.332982+0000","last_became_active":"2024-04-19T02:03:02.570116+0000","last_became_peered":"2024-04-19T02:03:02.570116+0000","last_unstale":"2024-04-19T02:04:39.332982+0000","last_undegraded":"2024-04-19T02:04:39.332982+0000","last_fullsized":"2024-04-19T02:04:39.332982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:25.439289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:25.439289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:28.101891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":19,"seq":81604378649,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32492,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339348,"statfs":{"total":96636764160,"available":96603492352,"internally_reserved":0,"allocated":1343488,"data_stored":835980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":19,"seq":81604378648,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411352,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50552,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321288,"statfs":{"total":96636764160,"available":96584998912,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411353,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50552,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321288,"statfs":{"total":96636764160,"available":96584998912,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411353,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51076,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320764,"statfs":{"total":96636764160,"available":96584462336,"internally_reserved":0,"allocated":1888256,"data_stored":1372608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411353,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50556,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321284,"statfs":{"total":96636764160,"available":96584994816,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":18,"seq":77309411351,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28332,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343508,"statfs":{"total":96636764160,"available":96607752192,"internally_reserved":0,"allocated":1343488,"data_stored":835980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411352,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32508,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":18,"seq":77309411352,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50556,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321284,"statfs":{"total":96636764160,"available":96584994816,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":17,"seq":73014444056,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32512,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":17,"seq":73014444056,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51072,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320768,"statfs":{"total":96636764160,"available":96584466432,"internally_reserved":0,"allocated":1875968,"data_stored":1364908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":17,"seq":73014444056,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50560,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321280,"statfs":{"total":96636764160,"available":96584990720,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T02:04:43.717 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-19T02:04:43.717 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-19T02:04:43.728 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-19T02:04:43.728 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T02:04:44.011 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:04:44.012 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-19T02:04:44.028 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-04-19T02:04:42.233031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":513280,"kb_used_data":16892,"kb_used_omap":18,"kb_used_meta":496173,"kb_avail":1131948800,"statfs":{"total":1159641169920,"available":1159115571200,"internally_reserved":0,"allocated":17297408,"data_stored":11158916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":508081465},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.939715"},"pg_stats":[{"pgid":"1.0","version":"96'139","reported_seq":384,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.333126+0000","last_change":"2024-04-19T02:04:20.382306+0000","last_active":"2024-04-19T02:04:39.333126+0000","last_peered":"2024-04-19T02:04:39.333126+0000","last_clean":"2024-04-19T02:04:39.333126+0000","last_became_active":"2024-04-19T02:02:59.972005+0000","last_became_peered":"2024-04-19T02:02:59.972005+0000","last_unstale":"2024-04-19T02:04:39.333126+0000","last_undegraded":"2024-04-19T02:04:39.333126+0000","last_fullsized":"2024-04-19T02:04:39.333126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'139","last_scrub_stamp":"2024-04-19T02:04:20.382039+0000","last_deep_scrub":"96'139","last_deep_scrub_stamp":"2024-04-19T02:04:20.382039+0000","last_clean_scrub_stamp":"2024-04-19T02:04:20.382039+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:42.019594+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":201,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.723615+0000","last_change":"2024-04-19T02:04:22.026870+0000","last_active":"2024-04-19T02:04:39.723615+0000","last_peered":"2024-04-19T02:04:39.723615+0000","last_clean":"2024-04-19T02:04:39.723615+0000","last_became_active":"2024-04-19T02:03:03.026063+0000","last_became_peered":"2024-04-19T02:03:03.026063+0000","last_unstale":"2024-04-19T02:04:39.723615+0000","last_undegraded":"2024-04-19T02:04:39.723615+0000","last_fullsized":"2024-04-19T02:04:39.723615+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:22.026513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:22.026513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:46.328574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":203,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.332912+0000","last_change":"2024-04-19T02:04:17.332692+0000","last_active":"2024-04-19T02:04:39.332912+0000","last_peered":"2024-04-19T02:04:39.332912+0000","last_clean":"2024-04-19T02:04:39.332912+0000","last_became_active":"2024-04-19T02:03:02.560210+0000","last_became_peered":"2024-04-19T02:03:02.560210+0000","last_unstale":"2024-04-19T02:04:39.332912+0000","last_undegraded":"2024-04-19T02:04:39.332912+0000","last_fullsized":"2024-04-19T02:04:39.332912+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:17.332474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:17.332474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:44.300211+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":60,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.722031+0000","last_change":"2024-04-19T02:04:26.463553+0000","last_active":"2024-04-19T02:04:39.722031+0000","last_peered":"2024-04-19T02:04:39.722031+0000","last_clean":"2024-04-19T02:04:39.722031+0000","last_became_active":"2024-04-19T02:04:22.996729+0000","last_became_peered":"2024-04-19T02:04:22.996729+0000","last_unstale":"2024-04-19T02:04:39.722031+0000","last_undegraded":"2024-04-19T02:04:39.722031+0000","last_fullsized":"2024-04-19T02:04:39.722031+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:26.463421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:26.463421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:33.227058+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":203,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.332953+0000","last_change":"2024-04-19T02:04:33.441354+0000","last_active":"2024-04-19T02:04:39.332953+0000","last_peered":"2024-04-19T02:04:39.332953+0000","last_clean":"2024-04-19T02:04:39.332953+0000","last_became_active":"2024-04-19T02:03:02.559377+0000","last_became_peered":"2024-04-19T02:03:02.559377+0000","last_unstale":"2024-04-19T02:04:39.332953+0000","last_undegraded":"2024-04-19T02:04:39.332953+0000","last_fullsized":"2024-04-19T02:04:39.332953+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:33.441118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:33.441118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:36.624187+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":201,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.332093+0000","last_change":"2024-04-19T02:04:16.610529+0000","last_active":"2024-04-19T02:04:39.332093+0000","last_peered":"2024-04-19T02:04:39.332093+0000","last_clean":"2024-04-19T02:04:39.332093+0000","last_became_active":"2024-04-19T02:03:02.597118+0000","last_became_peered":"2024-04-19T02:03:02.597118+0000","last_unstale":"2024-04-19T02:04:39.332093+0000","last_undegraded":"2024-04-19T02:04:39.332093+0000","last_fullsized":"2024-04-19T02:04:39.332093+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:16.610198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:16.610198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:35.025165+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":203,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.724014+0000","last_change":"2024-04-19T02:04:04.753289+0000","last_active":"2024-04-19T02:04:39.724014+0000","last_peered":"2024-04-19T02:04:39.724014+0000","last_clean":"2024-04-19T02:04:39.724014+0000","last_became_active":"2024-04-19T02:03:02.595232+0000","last_became_peered":"2024-04-19T02:03:02.595232+0000","last_unstale":"2024-04-19T02:04:39.724014+0000","last_undegraded":"2024-04-19T02:04:39.724014+0000","last_fullsized":"2024-04-19T02:04:39.724014+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:04.752922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:04.752922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:25.635349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":201,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.332074+0000","last_change":"2024-04-19T02:04:29.771190+0000","last_active":"2024-04-19T02:04:39.332074+0000","last_peered":"2024-04-19T02:04:39.332074+0000","last_clean":"2024-04-19T02:04:39.332074+0000","last_became_active":"2024-04-19T02:03:02.988073+0000","last_became_peered":"2024-04-19T02:03:02.988073+0000","last_unstale":"2024-04-19T02:04:39.332074+0000","last_undegraded":"2024-04-19T02:04:39.332074+0000","last_fullsized":"2024-04-19T02:04:39.332074+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:29.770959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:29.770959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:54.167208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":203,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.332982+0000","last_change":"2024-04-19T02:04:25.439628+0000","last_active":"2024-04-19T02:04:39.332982+0000","last_peered":"2024-04-19T02:04:39.332982+0000","last_clean":"2024-04-19T02:04:39.332982+0000","last_became_active":"2024-04-19T02:03:02.570116+0000","last_became_peered":"2024-04-19T02:03:02.570116+0000","last_unstale":"2024-04-19T02:04:39.332982+0000","last_undegraded":"2024-04-19T02:04:39.332982+0000","last_fullsized":"2024-04-19T02:04:39.332982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:25.439289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:25.439289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:28.101891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":19,"seq":81604378649,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32492,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339348,"statfs":{"total":96636764160,"available":96603492352,"internally_reserved":0,"allocated":1343488,"data_stored":835980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":19,"seq":81604378648,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411352,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50552,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321288,"statfs":{"total":96636764160,"available":96584998912,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411353,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50552,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321288,"statfs":{"total":96636764160,"available":96584998912,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411353,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51076,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320764,"statfs":{"total":96636764160,"available":96584462336,"internally_reserved":0,"allocated":1888256,"data_stored":1372608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411353,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50556,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321284,"statfs":{"total":96636764160,"available":96584994816,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":18,"seq":77309411351,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28332,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343508,"statfs":{"total":96636764160,"available":96607752192,"internally_reserved":0,"allocated":1343488,"data_stored":835980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411352,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32508,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":18,"seq":77309411352,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50556,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321284,"statfs":{"total":96636764160,"available":96584994816,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":17,"seq":73014444056,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32512,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":17,"seq":73014444056,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51072,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320768,"statfs":{"total":96636764160,"available":96584466432,"internally_reserved":0,"allocated":1875968,"data_stored":1364908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":17,"seq":73014444056,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50560,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321280,"statfs":{"total":96636764160,"available":96584990720,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T02:04:44.029 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T02:04:44.307 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:04:44.308 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-19T02:04:44.324 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2024-04-19T02:04:44.233647+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":513280,"kb_used_data":16904,"kb_used_omap":18,"kb_used_meta":496173,"kb_avail":1131948800,"statfs":{"total":1159641169920,"available":1159115571200,"internally_reserved":0,"allocated":17309696,"data_stored":11166616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":508081465},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.929966"},"pg_stats":[{"pgid":"1.0","version":"96'139","reported_seq":384,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.333126+0000","last_change":"2024-04-19T02:04:20.382306+0000","last_active":"2024-04-19T02:04:39.333126+0000","last_peered":"2024-04-19T02:04:39.333126+0000","last_clean":"2024-04-19T02:04:39.333126+0000","last_became_active":"2024-04-19T02:02:59.972005+0000","last_became_peered":"2024-04-19T02:02:59.972005+0000","last_unstale":"2024-04-19T02:04:39.333126+0000","last_undegraded":"2024-04-19T02:04:39.333126+0000","last_fullsized":"2024-04-19T02:04:39.333126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'139","last_scrub_stamp":"2024-04-19T02:04:20.382039+0000","last_deep_scrub":"96'139","last_deep_scrub_stamp":"2024-04-19T02:04:20.382039+0000","last_clean_scrub_stamp":"2024-04-19T02:04:20.382039+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:42.019594+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":201,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.723615+0000","last_change":"2024-04-19T02:04:22.026870+0000","last_active":"2024-04-19T02:04:39.723615+0000","last_peered":"2024-04-19T02:04:39.723615+0000","last_clean":"2024-04-19T02:04:39.723615+0000","last_became_active":"2024-04-19T02:03:03.026063+0000","last_became_peered":"2024-04-19T02:03:03.026063+0000","last_unstale":"2024-04-19T02:04:39.723615+0000","last_undegraded":"2024-04-19T02:04:39.723615+0000","last_fullsized":"2024-04-19T02:04:39.723615+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:22.026513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:22.026513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:46.328574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":203,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.332912+0000","last_change":"2024-04-19T02:04:17.332692+0000","last_active":"2024-04-19T02:04:39.332912+0000","last_peered":"2024-04-19T02:04:39.332912+0000","last_clean":"2024-04-19T02:04:39.332912+0000","last_became_active":"2024-04-19T02:03:02.560210+0000","last_became_peered":"2024-04-19T02:03:02.560210+0000","last_unstale":"2024-04-19T02:04:39.332912+0000","last_undegraded":"2024-04-19T02:04:39.332912+0000","last_fullsized":"2024-04-19T02:04:39.332912+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:17.332474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:17.332474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:44.300211+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":60,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.722031+0000","last_change":"2024-04-19T02:04:26.463553+0000","last_active":"2024-04-19T02:04:39.722031+0000","last_peered":"2024-04-19T02:04:39.722031+0000","last_clean":"2024-04-19T02:04:39.722031+0000","last_became_active":"2024-04-19T02:04:22.996729+0000","last_became_peered":"2024-04-19T02:04:22.996729+0000","last_unstale":"2024-04-19T02:04:39.722031+0000","last_undegraded":"2024-04-19T02:04:39.722031+0000","last_fullsized":"2024-04-19T02:04:39.722031+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:26.463421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:26.463421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:33.227058+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":203,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.332953+0000","last_change":"2024-04-19T02:04:33.441354+0000","last_active":"2024-04-19T02:04:39.332953+0000","last_peered":"2024-04-19T02:04:39.332953+0000","last_clean":"2024-04-19T02:04:39.332953+0000","last_became_active":"2024-04-19T02:03:02.559377+0000","last_became_peered":"2024-04-19T02:03:02.559377+0000","last_unstale":"2024-04-19T02:04:39.332953+0000","last_undegraded":"2024-04-19T02:04:39.332953+0000","last_fullsized":"2024-04-19T02:04:39.332953+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:33.441118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:33.441118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:36.624187+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":201,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.332093+0000","last_change":"2024-04-19T02:04:16.610529+0000","last_active":"2024-04-19T02:04:39.332093+0000","last_peered":"2024-04-19T02:04:39.332093+0000","last_clean":"2024-04-19T02:04:39.332093+0000","last_became_active":"2024-04-19T02:03:02.597118+0000","last_became_peered":"2024-04-19T02:03:02.597118+0000","last_unstale":"2024-04-19T02:04:39.332093+0000","last_undegraded":"2024-04-19T02:04:39.332093+0000","last_fullsized":"2024-04-19T02:04:39.332093+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:16.610198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:16.610198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:35.025165+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":203,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.724014+0000","last_change":"2024-04-19T02:04:04.753289+0000","last_active":"2024-04-19T02:04:39.724014+0000","last_peered":"2024-04-19T02:04:39.724014+0000","last_clean":"2024-04-19T02:04:39.724014+0000","last_became_active":"2024-04-19T02:03:02.595232+0000","last_became_peered":"2024-04-19T02:03:02.595232+0000","last_unstale":"2024-04-19T02:04:39.724014+0000","last_undegraded":"2024-04-19T02:04:39.724014+0000","last_fullsized":"2024-04-19T02:04:39.724014+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:04.752922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:04.752922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:25.635349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":201,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.332074+0000","last_change":"2024-04-19T02:04:29.771190+0000","last_active":"2024-04-19T02:04:39.332074+0000","last_peered":"2024-04-19T02:04:39.332074+0000","last_clean":"2024-04-19T02:04:39.332074+0000","last_became_active":"2024-04-19T02:03:02.988073+0000","last_became_peered":"2024-04-19T02:03:02.988073+0000","last_unstale":"2024-04-19T02:04:39.332074+0000","last_undegraded":"2024-04-19T02:04:39.332074+0000","last_fullsized":"2024-04-19T02:04:39.332074+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:29.770959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:29.770959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:54.167208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":203,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.332982+0000","last_change":"2024-04-19T02:04:25.439628+0000","last_active":"2024-04-19T02:04:39.332982+0000","last_peered":"2024-04-19T02:04:39.332982+0000","last_clean":"2024-04-19T02:04:39.332982+0000","last_became_active":"2024-04-19T02:03:02.570116+0000","last_became_peered":"2024-04-19T02:03:02.570116+0000","last_unstale":"2024-04-19T02:04:39.332982+0000","last_undegraded":"2024-04-19T02:04:39.332982+0000","last_fullsized":"2024-04-19T02:04:39.332982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:25.439289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:25.439289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:28.101891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":19,"seq":81604378649,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32492,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339348,"statfs":{"total":96636764160,"available":96603492352,"internally_reserved":0,"allocated":1343488,"data_stored":835980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":19,"seq":81604378648,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411352,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50552,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321288,"statfs":{"total":96636764160,"available":96584998912,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411353,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50552,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321288,"statfs":{"total":96636764160,"available":96584998912,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411353,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51076,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320764,"statfs":{"total":96636764160,"available":96584462336,"internally_reserved":0,"allocated":1888256,"data_stored":1372608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411353,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50556,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321284,"statfs":{"total":96636764160,"available":96584994816,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":18,"seq":77309411351,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28332,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343508,"statfs":{"total":96636764160,"available":96607752192,"internally_reserved":0,"allocated":1343488,"data_stored":835980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411352,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32508,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":18,"seq":77309411352,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50556,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321284,"statfs":{"total":96636764160,"available":96584994816,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":17,"seq":73014444057,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32512,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":17,"seq":73014444057,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51072,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320768,"statfs":{"total":96636764160,"available":96584466432,"internally_reserved":0,"allocated":1888256,"data_stored":1372608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":17,"seq":73014444057,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50560,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321280,"statfs":{"total":96636764160,"available":96584990720,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T02:04:44.325 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-19T02:04:44.326 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T02:04:44.622 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:04:44.623 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-19T02:04:44.639 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2024-04-19T02:04:44.233647+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":513280,"kb_used_data":16904,"kb_used_omap":18,"kb_used_meta":496173,"kb_avail":1131948800,"statfs":{"total":1159641169920,"available":1159115571200,"internally_reserved":0,"allocated":17309696,"data_stored":11166616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":508081465},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.929966"},"pg_stats":[{"pgid":"1.0","version":"96'139","reported_seq":384,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.333126+0000","last_change":"2024-04-19T02:04:20.382306+0000","last_active":"2024-04-19T02:04:39.333126+0000","last_peered":"2024-04-19T02:04:39.333126+0000","last_clean":"2024-04-19T02:04:39.333126+0000","last_became_active":"2024-04-19T02:02:59.972005+0000","last_became_peered":"2024-04-19T02:02:59.972005+0000","last_unstale":"2024-04-19T02:04:39.333126+0000","last_undegraded":"2024-04-19T02:04:39.333126+0000","last_fullsized":"2024-04-19T02:04:39.333126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'139","last_scrub_stamp":"2024-04-19T02:04:20.382039+0000","last_deep_scrub":"96'139","last_deep_scrub_stamp":"2024-04-19T02:04:20.382039+0000","last_clean_scrub_stamp":"2024-04-19T02:04:20.382039+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:42.019594+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":201,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.723615+0000","last_change":"2024-04-19T02:04:22.026870+0000","last_active":"2024-04-19T02:04:39.723615+0000","last_peered":"2024-04-19T02:04:39.723615+0000","last_clean":"2024-04-19T02:04:39.723615+0000","last_became_active":"2024-04-19T02:03:03.026063+0000","last_became_peered":"2024-04-19T02:03:03.026063+0000","last_unstale":"2024-04-19T02:04:39.723615+0000","last_undegraded":"2024-04-19T02:04:39.723615+0000","last_fullsized":"2024-04-19T02:04:39.723615+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:22.026513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:22.026513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:46.328574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":203,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.332912+0000","last_change":"2024-04-19T02:04:17.332692+0000","last_active":"2024-04-19T02:04:39.332912+0000","last_peered":"2024-04-19T02:04:39.332912+0000","last_clean":"2024-04-19T02:04:39.332912+0000","last_became_active":"2024-04-19T02:03:02.560210+0000","last_became_peered":"2024-04-19T02:03:02.560210+0000","last_unstale":"2024-04-19T02:04:39.332912+0000","last_undegraded":"2024-04-19T02:04:39.332912+0000","last_fullsized":"2024-04-19T02:04:39.332912+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:17.332474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:17.332474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:44.300211+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":60,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.722031+0000","last_change":"2024-04-19T02:04:26.463553+0000","last_active":"2024-04-19T02:04:39.722031+0000","last_peered":"2024-04-19T02:04:39.722031+0000","last_clean":"2024-04-19T02:04:39.722031+0000","last_became_active":"2024-04-19T02:04:22.996729+0000","last_became_peered":"2024-04-19T02:04:22.996729+0000","last_unstale":"2024-04-19T02:04:39.722031+0000","last_undegraded":"2024-04-19T02:04:39.722031+0000","last_fullsized":"2024-04-19T02:04:39.722031+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:26.463421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:26.463421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:33.227058+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":203,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.332953+0000","last_change":"2024-04-19T02:04:33.441354+0000","last_active":"2024-04-19T02:04:39.332953+0000","last_peered":"2024-04-19T02:04:39.332953+0000","last_clean":"2024-04-19T02:04:39.332953+0000","last_became_active":"2024-04-19T02:03:02.559377+0000","last_became_peered":"2024-04-19T02:03:02.559377+0000","last_unstale":"2024-04-19T02:04:39.332953+0000","last_undegraded":"2024-04-19T02:04:39.332953+0000","last_fullsized":"2024-04-19T02:04:39.332953+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:33.441118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:33.441118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:36.624187+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":201,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.332093+0000","last_change":"2024-04-19T02:04:16.610529+0000","last_active":"2024-04-19T02:04:39.332093+0000","last_peered":"2024-04-19T02:04:39.332093+0000","last_clean":"2024-04-19T02:04:39.332093+0000","last_became_active":"2024-04-19T02:03:02.597118+0000","last_became_peered":"2024-04-19T02:03:02.597118+0000","last_unstale":"2024-04-19T02:04:39.332093+0000","last_undegraded":"2024-04-19T02:04:39.332093+0000","last_fullsized":"2024-04-19T02:04:39.332093+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:16.610198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:16.610198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:35.025165+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":203,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.724014+0000","last_change":"2024-04-19T02:04:04.753289+0000","last_active":"2024-04-19T02:04:39.724014+0000","last_peered":"2024-04-19T02:04:39.724014+0000","last_clean":"2024-04-19T02:04:39.724014+0000","last_became_active":"2024-04-19T02:03:02.595232+0000","last_became_peered":"2024-04-19T02:03:02.595232+0000","last_unstale":"2024-04-19T02:04:39.724014+0000","last_undegraded":"2024-04-19T02:04:39.724014+0000","last_fullsized":"2024-04-19T02:04:39.724014+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:04.752922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:04.752922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:25.635349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":201,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.332074+0000","last_change":"2024-04-19T02:04:29.771190+0000","last_active":"2024-04-19T02:04:39.332074+0000","last_peered":"2024-04-19T02:04:39.332074+0000","last_clean":"2024-04-19T02:04:39.332074+0000","last_became_active":"2024-04-19T02:03:02.988073+0000","last_became_peered":"2024-04-19T02:03:02.988073+0000","last_unstale":"2024-04-19T02:04:39.332074+0000","last_undegraded":"2024-04-19T02:04:39.332074+0000","last_fullsized":"2024-04-19T02:04:39.332074+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:29.770959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:29.770959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:54.167208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":203,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.332982+0000","last_change":"2024-04-19T02:04:25.439628+0000","last_active":"2024-04-19T02:04:39.332982+0000","last_peered":"2024-04-19T02:04:39.332982+0000","last_clean":"2024-04-19T02:04:39.332982+0000","last_became_active":"2024-04-19T02:03:02.570116+0000","last_became_peered":"2024-04-19T02:03:02.570116+0000","last_unstale":"2024-04-19T02:04:39.332982+0000","last_undegraded":"2024-04-19T02:04:39.332982+0000","last_fullsized":"2024-04-19T02:04:39.332982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:25.439289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:25.439289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:28.101891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":19,"seq":81604378649,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32492,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339348,"statfs":{"total":96636764160,"available":96603492352,"internally_reserved":0,"allocated":1343488,"data_stored":835980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":19,"seq":81604378648,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411352,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50552,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321288,"statfs":{"total":96636764160,"available":96584998912,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411353,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50552,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321288,"statfs":{"total":96636764160,"available":96584998912,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411353,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51076,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320764,"statfs":{"total":96636764160,"available":96584462336,"internally_reserved":0,"allocated":1888256,"data_stored":1372608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411353,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50556,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321284,"statfs":{"total":96636764160,"available":96584994816,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":18,"seq":77309411351,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28332,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343508,"statfs":{"total":96636764160,"available":96607752192,"internally_reserved":0,"allocated":1343488,"data_stored":835980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411352,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32508,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":18,"seq":77309411352,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50556,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321284,"statfs":{"total":96636764160,"available":96584994816,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":17,"seq":73014444057,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32512,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":17,"seq":73014444057,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51072,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320768,"statfs":{"total":96636764160,"available":96584466432,"internally_reserved":0,"allocated":1888256,"data_stored":1372608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":17,"seq":73014444057,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50560,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321280,"statfs":{"total":96636764160,"available":96584990720,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T02:04:44.640 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-19T02:04:44.936 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:04:44.936 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":115,"fsid":"db9b8c36-b038-40a6-b749-a60ca7443e45","created":"2024-04-19T02:02:47.352143+0000","modified":"2024-04-19T02:04:39.321038+0000","last_up_change":"2024-04-19T02:02:59.513773+0000","last_in_change":"2024-04-19T02:02:51.039271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-19T02:02:58.287078+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-19T02:03:00.613285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.5199999809265137,"score_stable":4.5199999809265137,"optimal_score":0.82999998331069946,"raw_score_acting":3.75,"raw_score_stable":3.75,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dff2e9d5-c79f-4cc8-b68c-d1d35cc3dfeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1145933874},{"type":"v1","addr":"172.21.15.59:6809","nonce":1145933874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1145933874},{"type":"v1","addr":"172.21.15.59:6811","nonce":1145933874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1145933874},{"type":"v1","addr":"172.21.15.59:6815","nonce":1145933874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1145933874},{"type":"v1","addr":"172.21.15.59:6813","nonce":1145933874}]},"public_addr":"172.21.15.59:6809/1145933874","cluster_addr":"172.21.15.59:6811/1145933874","heartbeat_back_addr":"172.21.15.59:6815/1145933874","heartbeat_front_addr":"172.21.15.59:6813/1145933874","state":["exists","up"]},{"osd":1,"uuid":"30cb7aa1-0bf8-4bc8-8bc5-f8b59870624e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":3807848039},{"type":"v1","addr":"172.21.15.59:6801","nonce":3807848039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":3807848039},{"type":"v1","addr":"172.21.15.59:6803","nonce":3807848039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":3807848039},{"type":"v1","addr":"172.21.15.59:6807","nonce":3807848039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":3807848039},{"type":"v1","addr":"172.21.15.59:6805","nonce":3807848039}]},"public_addr":"172.21.15.59:6801/3807848039","cluster_addr":"172.21.15.59:6803/3807848039","heartbeat_back_addr":"172.21.15.59:6807/3807848039","heartbeat_front_addr":"172.21.15.59:6805/3807848039","state":["exists","up"]},{"osd":2,"uuid":"2e8fca1b-e008-4e58-865e-09a658f6edd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":420007638},{"type":"v1","addr":"172.21.15.59:6817","nonce":420007638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":420007638},{"type":"v1","addr":"172.21.15.59:6822","nonce":420007638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6829","nonce":420007638},{"type":"v1","addr":"172.21.15.59:6831","nonce":420007638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":420007638},{"type":"v1","addr":"172.21.15.59:6827","nonce":420007638}]},"public_addr":"172.21.15.59:6817/420007638","cluster_addr":"172.21.15.59:6822/420007638","heartbeat_back_addr":"172.21.15.59:6831/420007638","heartbeat_front_addr":"172.21.15.59:6827/420007638","state":["exists","up"]},{"osd":3,"uuid":"32289c9a-b7d9-4bcd-9f66-fa68f0c9796b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1791539182},{"type":"v1","addr":"172.21.15.59:6819","nonce":1791539182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6821","nonce":1791539182},{"type":"v1","addr":"172.21.15.59:6823","nonce":1791539182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1791539182},{"type":"v1","addr":"172.21.15.59:6830","nonce":1791539182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6825","nonce":1791539182},{"type":"v1","addr":"172.21.15.59:6826","nonce":1791539182}]},"public_addr":"172.21.15.59:6819/1791539182","cluster_addr":"172.21.15.59:6823/1791539182","heartbeat_back_addr":"172.21.15.59:6830/1791539182","heartbeat_front_addr":"172.21.15.59:6826/1791539182","state":["exists","up"]},{"osd":4,"uuid":"844daeea-860b-4853-a759-994531e5de3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":2921507000},{"type":"v1","addr":"172.21.15.81:6808","nonce":2921507000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":2921507000},{"type":"v1","addr":"172.21.15.81:6811","nonce":2921507000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":2921507000},{"type":"v1","addr":"172.21.15.81:6815","nonce":2921507000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2921507000},{"type":"v1","addr":"172.21.15.81:6813","nonce":2921507000}]},"public_addr":"172.21.15.81:6808/2921507000","cluster_addr":"172.21.15.81:6811/2921507000","heartbeat_back_addr":"172.21.15.81:6815/2921507000","heartbeat_front_addr":"172.21.15.81:6813/2921507000","state":["exists","up"]},{"osd":5,"uuid":"f43ad412-74bb-43a6-95e7-25ee06bde025","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3925467179},{"type":"v1","addr":"172.21.15.81:6818","nonce":3925467179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":3925467179},{"type":"v1","addr":"172.21.15.81:6821","nonce":3925467179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":3925467179},{"type":"v1","addr":"172.21.15.81:6826","nonce":3925467179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":3925467179},{"type":"v1","addr":"172.21.15.81:6824","nonce":3925467179}]},"public_addr":"172.21.15.81:6818/3925467179","cluster_addr":"172.21.15.81:6821/3925467179","heartbeat_back_addr":"172.21.15.81:6826/3925467179","heartbeat_front_addr":"172.21.15.81:6824/3925467179","state":["exists","up"]},{"osd":6,"uuid":"07ccbc36-1f5d-4595-8578-09362b5e212f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3714790214},{"type":"v1","addr":"172.21.15.81:6801","nonce":3714790214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3714790214},{"type":"v1","addr":"172.21.15.81:6803","nonce":3714790214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6807","nonce":3714790214},{"type":"v1","addr":"172.21.15.81:6809","nonce":3714790214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3714790214},{"type":"v1","addr":"172.21.15.81:6805","nonce":3714790214}]},"public_addr":"172.21.15.81:6801/3714790214","cluster_addr":"172.21.15.81:6803/3714790214","heartbeat_back_addr":"172.21.15.81:6809/3714790214","heartbeat_front_addr":"172.21.15.81:6805/3714790214","state":["exists","up"]},{"osd":7,"uuid":"6b2247e7-ebf8-4c0e-89ae-e15985f59aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":2170932617},{"type":"v1","addr":"172.21.15.81:6820","nonce":2170932617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":2170932617},{"type":"v1","addr":"172.21.15.81:6827","nonce":2170932617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":2170932617},{"type":"v1","addr":"172.21.15.81:6831","nonce":2170932617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":2170932617},{"type":"v1","addr":"172.21.15.81:6829","nonce":2170932617}]},"public_addr":"172.21.15.81:6820/2170932617","cluster_addr":"172.21.15.81:6827/2170932617","heartbeat_back_addr":"172.21.15.81:6831/2170932617","heartbeat_front_addr":"172.21.15.81:6829/2170932617","state":["exists","up"]},{"osd":8,"uuid":"7142efd0-2144-4973-a01d-69a064b9b51f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2053612068},{"type":"v1","addr":"172.21.15.179:6825","nonce":2053612068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2053612068},{"type":"v1","addr":"172.21.15.179:6827","nonce":2053612068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2053612068},{"type":"v1","addr":"172.21.15.179:6831","nonce":2053612068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2053612068},{"type":"v1","addr":"172.21.15.179:6829","nonce":2053612068}]},"public_addr":"172.21.15.179:6825/2053612068","cluster_addr":"172.21.15.179:6827/2053612068","heartbeat_back_addr":"172.21.15.179:6831/2053612068","heartbeat_front_addr":"172.21.15.179:6829/2053612068","state":["exists","up"]},{"osd":9,"uuid":"30a29baa-53cc-4c29-ab41-af14ead5c539","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2641050728},{"type":"v1","addr":"172.21.15.179:6801","nonce":2641050728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2641050728},{"type":"v1","addr":"172.21.15.179:6803","nonce":2641050728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2641050728},{"type":"v1","addr":"172.21.15.179:6807","nonce":2641050728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2641050728},{"type":"v1","addr":"172.21.15.179:6805","nonce":2641050728}]},"public_addr":"172.21.15.179:6801/2641050728","cluster_addr":"172.21.15.179:6803/2641050728","heartbeat_back_addr":"172.21.15.179:6807/2641050728","heartbeat_front_addr":"172.21.15.179:6805/2641050728","state":["exists","up"]},{"osd":10,"uuid":"fae84343-8b7a-4bd3-aeef-823f2f6058e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3465727116},{"type":"v1","addr":"172.21.15.179:6817","nonce":3465727116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3465727116},{"type":"v1","addr":"172.21.15.179:6819","nonce":3465727116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3465727116},{"type":"v1","addr":"172.21.15.179:6823","nonce":3465727116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3465727116},{"type":"v1","addr":"172.21.15.179:6821","nonce":3465727116}]},"public_addr":"172.21.15.179:6817/3465727116","cluster_addr":"172.21.15.179:6819/3465727116","heartbeat_back_addr":"172.21.15.179:6823/3465727116","heartbeat_front_addr":"172.21.15.179:6821/3465727116","state":["exists","up"]},{"osd":11,"uuid":"35858134-6864-4da1-88c2-4b48166febab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":4047209718},{"type":"v1","addr":"172.21.15.179:6809","nonce":4047209718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":4047209718},{"type":"v1","addr":"172.21.15.179:6811","nonce":4047209718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4047209718},{"type":"v1","addr":"172.21.15.179:6815","nonce":4047209718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4047209718},{"type":"v1","addr":"172.21.15.179:6813","nonce":4047209718}]},"public_addr":"172.21.15.179:6809/4047209718","cluster_addr":"172.21.15.179:6811/4047209718","heartbeat_back_addr":"172.21.15.179:6815/4047209718","heartbeat_front_addr":"172.21.15.179:6813/4047209718","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:12.165335+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:26.292747+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:16.779735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:14.023955+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:15.627221+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:15.527302+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:16.257148+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:12.903521+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:17.699941+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:23.768672+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:04:17.057934+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T02:03:57.837672+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lrcprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","l":"3","m":"2","name":"lrcprofile","plugin":"lrc"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":27,"length":1},{"begin":29,"length":1},{"begin":33,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T02:04:45.955 INFO:tasks.ceph:Scrubbing osd.0 2024-04-19T02:04:45.955 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-19T02:04:46.119 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:04:46.120 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T02:04:46.120 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:04:46.135 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-19T02:04:46.421 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 0 to deep-scrub 2024-04-19T02:04:46.438 INFO:tasks.ceph:Scrubbing osd.1 2024-04-19T02:04:46.439 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-19T02:04:46.576 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:04:46.577 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T02:04:46.577 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:04:46.591 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-19T02:04:46.876 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 1 to deep-scrub 2024-04-19T02:04:46.895 INFO:tasks.ceph:Scrubbing osd.2 2024-04-19T02:04:46.895 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-19T02:04:47.041 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:04:47.041 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T02:04:47.042 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:04:47.056 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-19T02:04:47.341 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 2 to deep-scrub 2024-04-19T02:04:47.359 INFO:tasks.ceph:Scrubbing osd.3 2024-04-19T02:04:47.359 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-19T02:04:47.510 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:04:47.511 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T02:04:47.511 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:04:47.527 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-19T02:04:47.812 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 3 to deep-scrub 2024-04-19T02:04:47.831 INFO:tasks.ceph:Scrubbing osd.4 2024-04-19T02:04:47.831 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-19T02:04:47.975 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:04:47.975 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T02:04:47.975 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:04:47.991 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-19T02:04:48.277 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 4 to deep-scrub 2024-04-19T02:04:48.296 INFO:tasks.ceph:Scrubbing osd.5 2024-04-19T02:04:48.296 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-19T02:04:48.453 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:04:48.453 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T02:04:48.453 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:04:48.469 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-19T02:04:48.768 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 5 to deep-scrub 2024-04-19T02:04:48.787 INFO:tasks.ceph:Scrubbing osd.6 2024-04-19T02:04:48.788 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-19T02:04:48.948 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:04:48.948 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T02:04:48.948 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:04:48.964 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-19T02:04:49.257 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 6 to deep-scrub 2024-04-19T02:04:49.276 INFO:tasks.ceph:Scrubbing osd.7 2024-04-19T02:04:49.276 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-19T02:04:49.426 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:04:49.427 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T02:04:49.427 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:04:49.443 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-19T02:04:49.728 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 7 to deep-scrub 2024-04-19T02:04:49.746 INFO:tasks.ceph:Scrubbing osd.8 2024-04-19T02:04:49.747 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 config set osd_debug_deep_scrub_sleep 0 2024-04-19T02:04:49.898 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:04:49.898 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T02:04:49.898 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:04:49.914 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 8 2024-04-19T02:04:50.195 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 8 to deep-scrub 2024-04-19T02:04:50.213 INFO:tasks.ceph:Scrubbing osd.9 2024-04-19T02:04:50.214 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 config set osd_debug_deep_scrub_sleep 0 2024-04-19T02:04:50.358 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:04:50.359 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T02:04:50.359 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:04:50.374 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 9 2024-04-19T02:04:50.669 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 9 to deep-scrub 2024-04-19T02:04:50.689 INFO:tasks.ceph:Scrubbing osd.10 2024-04-19T02:04:50.689 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 config set osd_debug_deep_scrub_sleep 0 2024-04-19T02:04:50.837 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:04:50.837 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T02:04:50.837 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:04:50.854 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 10 2024-04-19T02:04:51.141 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 10 to deep-scrub 2024-04-19T02:04:51.159 INFO:tasks.ceph:Scrubbing osd.11 2024-04-19T02:04:51.159 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 config set osd_debug_deep_scrub_sleep 0 2024-04-19T02:04:51.313 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-04-19T02:04:51.313 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T02:04:51.313 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-04-19T02:04:51.329 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 11 2024-04-19T02:04:51.612 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 11 to deep-scrub 2024-04-19T02:04:51.630 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T02:04:51.914 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:04:51.914 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-19T02:04:51.930 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-04-19T02:04:50.235341+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":513312,"kb_used_data":16928,"kb_used_omap":18,"kb_used_meta":496173,"kb_avail":1131948768,"statfs":{"total":1159641169920,"available":1159115538432,"internally_reserved":0,"allocated":17334272,"data_stored":11182016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":508081465},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.906015"},"pg_stats":[{"pgid":"1.0","version":"96'139","reported_seq":384,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.333126+0000","last_change":"2024-04-19T02:04:20.382306+0000","last_active":"2024-04-19T02:04:39.333126+0000","last_peered":"2024-04-19T02:04:39.333126+0000","last_clean":"2024-04-19T02:04:39.333126+0000","last_became_active":"2024-04-19T02:02:59.972005+0000","last_became_peered":"2024-04-19T02:02:59.972005+0000","last_unstale":"2024-04-19T02:04:39.333126+0000","last_undegraded":"2024-04-19T02:04:39.333126+0000","last_fullsized":"2024-04-19T02:04:39.333126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'139","last_scrub_stamp":"2024-04-19T02:04:20.382039+0000","last_deep_scrub":"96'139","last_deep_scrub_stamp":"2024-04-19T02:04:20.382039+0000","last_clean_scrub_stamp":"2024-04-19T02:04:20.382039+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:42.019594+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":201,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.723615+0000","last_change":"2024-04-19T02:04:22.026870+0000","last_active":"2024-04-19T02:04:39.723615+0000","last_peered":"2024-04-19T02:04:39.723615+0000","last_clean":"2024-04-19T02:04:39.723615+0000","last_became_active":"2024-04-19T02:03:03.026063+0000","last_became_peered":"2024-04-19T02:03:03.026063+0000","last_unstale":"2024-04-19T02:04:39.723615+0000","last_undegraded":"2024-04-19T02:04:39.723615+0000","last_fullsized":"2024-04-19T02:04:39.723615+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:22.026513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:22.026513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:46.328574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":203,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.332912+0000","last_change":"2024-04-19T02:04:17.332692+0000","last_active":"2024-04-19T02:04:39.332912+0000","last_peered":"2024-04-19T02:04:39.332912+0000","last_clean":"2024-04-19T02:04:39.332912+0000","last_became_active":"2024-04-19T02:03:02.560210+0000","last_became_peered":"2024-04-19T02:03:02.560210+0000","last_unstale":"2024-04-19T02:04:39.332912+0000","last_undegraded":"2024-04-19T02:04:39.332912+0000","last_fullsized":"2024-04-19T02:04:39.332912+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:17.332474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:17.332474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:44.300211+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":60,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.722031+0000","last_change":"2024-04-19T02:04:26.463553+0000","last_active":"2024-04-19T02:04:39.722031+0000","last_peered":"2024-04-19T02:04:39.722031+0000","last_clean":"2024-04-19T02:04:39.722031+0000","last_became_active":"2024-04-19T02:04:22.996729+0000","last_became_peered":"2024-04-19T02:04:22.996729+0000","last_unstale":"2024-04-19T02:04:39.722031+0000","last_undegraded":"2024-04-19T02:04:39.722031+0000","last_fullsized":"2024-04-19T02:04:39.722031+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:26.463421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:26.463421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:33.227058+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":203,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.332953+0000","last_change":"2024-04-19T02:04:33.441354+0000","last_active":"2024-04-19T02:04:39.332953+0000","last_peered":"2024-04-19T02:04:39.332953+0000","last_clean":"2024-04-19T02:04:39.332953+0000","last_became_active":"2024-04-19T02:03:02.559377+0000","last_became_peered":"2024-04-19T02:03:02.559377+0000","last_unstale":"2024-04-19T02:04:39.332953+0000","last_undegraded":"2024-04-19T02:04:39.332953+0000","last_fullsized":"2024-04-19T02:04:39.332953+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:33.441118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:33.441118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:36.624187+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":210,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:47.844280+0000","last_change":"2024-04-19T02:04:47.844301+0000","last_active":"2024-04-19T02:04:47.844280+0000","last_peered":"2024-04-19T02:04:47.844280+0000","last_clean":"2024-04-19T02:04:47.844280+0000","last_became_active":"2024-04-19T02:03:02.597118+0000","last_became_peered":"2024-04-19T02:03:02.597118+0000","last_unstale":"2024-04-19T02:04:47.844280+0000","last_undegraded":"2024-04-19T02:04:47.844280+0000","last_fullsized":"2024-04-19T02:04:47.844280+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:47.844195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:04:47.844195+0000","last_clean_scrub_stamp":"2024-04-19T02:04:47.844195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:55.829237+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":203,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.724014+0000","last_change":"2024-04-19T02:04:04.753289+0000","last_active":"2024-04-19T02:04:39.724014+0000","last_peered":"2024-04-19T02:04:39.724014+0000","last_clean":"2024-04-19T02:04:39.724014+0000","last_became_active":"2024-04-19T02:03:02.595232+0000","last_became_peered":"2024-04-19T02:03:02.595232+0000","last_unstale":"2024-04-19T02:04:39.724014+0000","last_undegraded":"2024-04-19T02:04:39.724014+0000","last_fullsized":"2024-04-19T02:04:39.724014+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:04.752922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:04.752922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:25.635349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":202,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.332124+0000","last_change":"2024-04-19T02:04:29.771190+0000","last_active":"2024-04-19T02:04:39.332124+0000","last_peered":"2024-04-19T02:04:39.332124+0000","last_clean":"2024-04-19T02:04:39.332124+0000","last_became_active":"2024-04-19T02:03:02.988073+0000","last_became_peered":"2024-04-19T02:03:02.988073+0000","last_unstale":"2024-04-19T02:04:39.332124+0000","last_undegraded":"2024-04-19T02:04:39.332124+0000","last_fullsized":"2024-04-19T02:04:39.332124+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:29.770959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:29.770959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":203,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-19T02:04:39.332982+0000","last_change":"2024-04-19T02:04:25.439628+0000","last_active":"2024-04-19T02:04:39.332982+0000","last_peered":"2024-04-19T02:04:39.332982+0000","last_clean":"2024-04-19T02:04:39.332982+0000","last_became_active":"2024-04-19T02:03:02.570116+0000","last_became_peered":"2024-04-19T02:03:02.570116+0000","last_unstale":"2024-04-19T02:04:39.332982+0000","last_undegraded":"2024-04-19T02:04:39.332982+0000","last_fullsized":"2024-04-19T02:04:39.332982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:25.439289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:03:01.531230+0000","last_clean_scrub_stamp":"2024-04-19T02:04:25.439289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:28.101891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":19,"seq":81604378650,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32508,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":19,"seq":81604378649,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411354,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50552,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321288,"statfs":{"total":96636764160,"available":96584998912,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411355,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50552,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321288,"statfs":{"total":96636764160,"available":96584998912,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411355,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51076,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320764,"statfs":{"total":96636764160,"available":96584462336,"internally_reserved":0,"allocated":1888256,"data_stored":1372608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411355,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50556,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321284,"statfs":{"total":96636764160,"available":96584994816,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":18,"seq":77309411353,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28348,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343492,"statfs":{"total":96636764160,"available":96607735808,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411354,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32508,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":18,"seq":77309411354,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50556,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321284,"statfs":{"total":96636764160,"available":96584994816,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":17,"seq":73014444058,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32512,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":17,"seq":73014444058,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51072,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320768,"statfs":{"total":96636764160,"available":96584466432,"internally_reserved":0,"allocated":1888256,"data_stored":1372608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":17,"seq":73014444058,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50560,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321280,"statfs":{"total":96636764160,"available":96584990720,"internally_reserved":0,"allocated":1355776,"data_stored":843680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T02:04:51.934 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-19T02:04:20.382039+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=2, tm_min=4, tm_sec=20, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=2, tm_min=4, tm_sec=44, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T02:04:51.934 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-19T02:04:22.026513+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=2, tm_min=4, tm_sec=22, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=2, tm_min=4, tm_sec=44, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T02:04:51.934 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-19T02:04:17.332474+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=2, tm_min=4, tm_sec=17, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=2, tm_min=4, tm_sec=44, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T02:04:51.934 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-19T02:04:26.463421+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=2, tm_min=4, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=2, tm_min=4, tm_sec=44, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T02:04:51.935 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-19T02:04:33.441118+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=2, tm_min=4, tm_sec=33, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=2, tm_min=4, tm_sec=44, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T02:04:51.935 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-19T02:04:04.752922+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=2, tm_min=4, tm_sec=4, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=2, tm_min=4, tm_sec=44, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T02:04:51.935 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-19T02:04:29.770959+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=2, tm_min=4, tm_sec=29, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=2, tm_min=4, tm_sec=44, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T02:04:51.935 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-19T02:04:25.439289+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=2, tm_min=4, tm_sec=25, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=2, tm_min=4, tm_sec=44, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T02:04:51.935 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-19T02:05:11.936 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T02:05:12.251 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-04-19T02:05:12.251 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-04-19T02:05:12.271 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2024-04-19T02:05:12.242859+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":513408,"kb_used_data":17072,"kb_used_omap":18,"kb_used_meta":496173,"kb_avail":1131948672,"statfs":{"total":1159641169920,"available":1159115440128,"internally_reserved":0,"allocated":17481728,"data_stored":11274920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":508081465},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003390"},"pg_stats":[{"pgid":"1.0","version":"96'139","reported_seq":396,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-19T02:05:05.440353+0000","last_change":"2024-04-19T02:04:50.386800+0000","last_active":"2024-04-19T02:05:05.440353+0000","last_peered":"2024-04-19T02:05:05.440353+0000","last_clean":"2024-04-19T02:05:05.440353+0000","last_became_active":"2024-04-19T02:02:59.972005+0000","last_became_peered":"2024-04-19T02:02:59.972005+0000","last_unstale":"2024-04-19T02:05:05.440353+0000","last_undegraded":"2024-04-19T02:05:05.440353+0000","last_fullsized":"2024-04-19T02:05:05.440353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'139","last_scrub_stamp":"2024-04-19T02:04:50.386401+0000","last_deep_scrub":"96'139","last_deep_scrub_stamp":"2024-04-19T02:04:50.386401+0000","last_clean_scrub_stamp":"2024-04-19T02:04:50.386401+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:06:07.027883+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":213,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-19T02:05:05.872638+0000","last_change":"2024-04-19T02:04:52.312210+0000","last_active":"2024-04-19T02:05:05.872638+0000","last_peered":"2024-04-19T02:05:05.872638+0000","last_clean":"2024-04-19T02:05:05.872638+0000","last_became_active":"2024-04-19T02:03:03.026063+0000","last_became_peered":"2024-04-19T02:03:03.026063+0000","last_unstale":"2024-04-19T02:05:05.872638+0000","last_undegraded":"2024-04-19T02:05:05.872638+0000","last_fullsized":"2024-04-19T02:05:05.872638+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:52.311977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:04:52.311977+0000","last_clean_scrub_stamp":"2024-04-19T02:04:52.311977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:06:19.735591+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":215,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-19T02:05:05.440315+0000","last_change":"2024-04-19T02:04:52.349340+0000","last_active":"2024-04-19T02:05:05.440315+0000","last_peered":"2024-04-19T02:05:05.440315+0000","last_clean":"2024-04-19T02:05:05.440315+0000","last_became_active":"2024-04-19T02:03:02.560210+0000","last_became_peered":"2024-04-19T02:03:02.560210+0000","last_unstale":"2024-04-19T02:05:05.440315+0000","last_undegraded":"2024-04-19T02:05:05.440315+0000","last_fullsized":"2024-04-19T02:05:05.440315+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:52.349018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:04:52.349018+0000","last_clean_scrub_stamp":"2024-04-19T02:04:52.349018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:06:05.907747+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":72,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-19T02:05:06.727289+0000","last_change":"2024-04-19T02:04:47.656182+0000","last_active":"2024-04-19T02:05:06.727289+0000","last_peered":"2024-04-19T02:05:06.727289+0000","last_clean":"2024-04-19T02:05:06.727289+0000","last_became_active":"2024-04-19T02:04:22.996729+0000","last_became_peered":"2024-04-19T02:04:22.996729+0000","last_unstale":"2024-04-19T02:05:06.727289+0000","last_undegraded":"2024-04-19T02:05:06.727289+0000","last_fullsized":"2024-04-19T02:05:06.727289+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:47.656082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:04:47.656082+0000","last_clean_scrub_stamp":"2024-04-19T02:04:47.656082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:49.888340+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":215,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-19T02:05:05.440441+0000","last_change":"2024-04-19T02:04:51.370079+0000","last_active":"2024-04-19T02:05:05.440441+0000","last_peered":"2024-04-19T02:05:05.440441+0000","last_clean":"2024-04-19T02:05:05.440441+0000","last_became_active":"2024-04-19T02:03:02.559377+0000","last_became_peered":"2024-04-19T02:03:02.559377+0000","last_unstale":"2024-04-19T02:05:05.440441+0000","last_undegraded":"2024-04-19T02:05:05.440441+0000","last_fullsized":"2024-04-19T02:05:05.440441+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:51.369834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:04:51.369834+0000","last_clean_scrub_stamp":"2024-04-19T02:04:51.369834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:06:15.342576+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":213,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-19T02:05:05.439099+0000","last_change":"2024-04-19T02:04:47.844301+0000","last_active":"2024-04-19T02:05:05.439099+0000","last_peered":"2024-04-19T02:05:05.439099+0000","last_clean":"2024-04-19T02:05:05.439099+0000","last_became_active":"2024-04-19T02:03:02.597118+0000","last_became_peered":"2024-04-19T02:03:02.597118+0000","last_unstale":"2024-04-19T02:05:05.439099+0000","last_undegraded":"2024-04-19T02:05:05.439099+0000","last_fullsized":"2024-04-19T02:05:05.439099+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:47.844195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:04:47.844195+0000","last_clean_scrub_stamp":"2024-04-19T02:04:47.844195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:55.829237+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":215,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-19T02:05:06.220770+0000","last_change":"2024-04-19T02:04:50.676549+0000","last_active":"2024-04-19T02:05:06.220770+0000","last_peered":"2024-04-19T02:05:06.220770+0000","last_clean":"2024-04-19T02:05:06.220770+0000","last_became_active":"2024-04-19T02:03:02.595232+0000","last_became_peered":"2024-04-19T02:03:02.595232+0000","last_unstale":"2024-04-19T02:05:06.220770+0000","last_undegraded":"2024-04-19T02:05:06.220770+0000","last_fullsized":"2024-04-19T02:05:06.220770+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:50.676324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:04:50.676324+0000","last_clean_scrub_stamp":"2024-04-19T02:04:50.676324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:56.202728+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":212,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-19T02:05:05.438955+0000","last_change":"2024-04-19T02:04:48.844196+0000","last_active":"2024-04-19T02:05:05.438955+0000","last_peered":"2024-04-19T02:05:05.438955+0000","last_clean":"2024-04-19T02:05:05.438955+0000","last_became_active":"2024-04-19T02:03:02.988073+0000","last_became_peered":"2024-04-19T02:03:02.988073+0000","last_unstale":"2024-04-19T02:05:05.438955+0000","last_undegraded":"2024-04-19T02:05:05.438955+0000","last_fullsized":"2024-04-19T02:05:05.438955+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:48.844090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:04:48.844090+0000","last_clean_scrub_stamp":"2024-04-19T02:04:48.844090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:05:59.813536+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":214,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-19T02:05:05.440331+0000","last_change":"2024-04-19T02:04:53.352966+0000","last_active":"2024-04-19T02:05:05.440331+0000","last_peered":"2024-04-19T02:05:05.440331+0000","last_clean":"2024-04-19T02:05:05.440331+0000","last_became_active":"2024-04-19T02:03:02.570116+0000","last_became_peered":"2024-04-19T02:03:02.570116+0000","last_unstale":"2024-04-19T02:05:05.440331+0000","last_undegraded":"2024-04-19T02:05:05.440331+0000","last_fullsized":"2024-04-19T02:05:05.440331+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T02:04:53.352732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T02:04:53.352732+0000","last_clean_scrub_stamp":"2024-04-19T02:04:53.352732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:06:01.538955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":19,"seq":81604378655,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32516,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339324,"statfs":{"total":96636764160,"available":96603467776,"internally_reserved":0,"allocated":1368064,"data_stored":851422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":19,"seq":81604378654,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32516,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339324,"statfs":{"total":96636764160,"available":96603467776,"internally_reserved":0,"allocated":1368064,"data_stored":851422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411358,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50564,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321276,"statfs":{"total":96636764160,"available":96584986624,"internally_reserved":0,"allocated":1368064,"data_stored":851422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411359,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50564,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321276,"statfs":{"total":96636764160,"available":96584986624,"internally_reserved":0,"allocated":1368064,"data_stored":851422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411359,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51084,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320756,"statfs":{"total":96636764160,"available":96584454144,"internally_reserved":0,"allocated":1900544,"data_stored":1380350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411359,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50564,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321276,"statfs":{"total":96636764160,"available":96584986624,"internally_reserved":0,"allocated":1368064,"data_stored":851422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":18,"seq":77309411357,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28356,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343484,"statfs":{"total":96636764160,"available":96607727616,"internally_reserved":0,"allocated":1368064,"data_stored":851422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411358,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32516,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339324,"statfs":{"total":96636764160,"available":96603467776,"internally_reserved":0,"allocated":1368064,"data_stored":851422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":18,"seq":77309411358,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50564,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321276,"statfs":{"total":96636764160,"available":96584986624,"internally_reserved":0,"allocated":1368064,"data_stored":851422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":17,"seq":73014444062,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32516,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339324,"statfs":{"total":96636764160,"available":96603467776,"internally_reserved":0,"allocated":1368064,"data_stored":851422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":17,"seq":73014444062,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51084,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320756,"statfs":{"total":96636764160,"available":96584454144,"internally_reserved":0,"allocated":1900544,"data_stored":1380350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":17,"seq":73014444062,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50564,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321276,"statfs":{"total":96636764160,"available":96584986624,"internally_reserved":0,"allocated":1368064,"data_stored":851422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T02:05:12.273 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-19T02:05:12.597 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-19T02:05:12.597 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-19T02:05:12.597 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-19T02:05:12.597 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T02:05:12.669 INFO:tasks.ceph.osd.0:Stopped 2024-04-19T02:05:12.669 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-19T02:05:12.669 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T02:05:12.757 INFO:tasks.ceph.osd.1:Stopped 2024-04-19T02:05:12.757 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-19T02:05:12.757 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T02:05:12.833 INFO:tasks.ceph.osd.2:Stopped 2024-04-19T02:05:12.833 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-19T02:05:12.833 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T02:05:12.922 INFO:tasks.ceph.osd.3:Stopped 2024-04-19T02:05:12.923 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-19T02:05:12.923 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T02:05:13.018 INFO:tasks.ceph.osd.4:Stopped 2024-04-19T02:05:13.018 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-19T02:05:13.019 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T02:05:16.864 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-04-19T02:05:19.121 INFO:tasks.ceph.osd.5:Stopped 2024-04-19T02:05:19.121 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-19T02:05:19.121 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T02:05:19.205 INFO:tasks.ceph.osd.6:Stopped 2024-04-19T02:05:19.205 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-19T02:05:19.205 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T02:05:19.300 INFO:tasks.ceph.osd.7:Stopped 2024-04-19T02:05:19.300 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2024-04-19T02:05:19.300 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T02:05:19.374 INFO:tasks.ceph.osd.8:Stopped 2024-04-19T02:05:19.374 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2024-04-19T02:05:19.374 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T02:05:19.461 INFO:tasks.ceph.osd.9:Stopped 2024-04-19T02:05:19.461 DEBUG:tasks.ceph.osd.10:waiting for process to exit 2024-04-19T02:05:19.461 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T02:05:22.370 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.10 is failed for ~0s 2024-04-19T02:05:22.370 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-04-19T02:05:25.563 INFO:tasks.ceph.osd.10:Stopped 2024-04-19T02:05:25.563 DEBUG:tasks.ceph.osd.11:waiting for process to exit 2024-04-19T02:05:25.563 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T02:05:27.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.11 is failed for ~0s 2024-04-19T02:05:27.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.10 has been restored 2024-04-19T02:05:31.665 INFO:tasks.ceph.osd.11:Stopped 2024-04-19T02:05:31.665 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-19T02:05:31.666 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-19T02:05:31.666 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T02:05:31.695 INFO:tasks.ceph.mgr.x:Stopped 2024-04-19T02:05:31.695 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-19T02:05:31.695 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-19T02:05:31.695 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T02:05:31.720 INFO:tasks.ceph.mon.a:Stopped 2024-04-19T02:05:31.720 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-19T02:05:31.720 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T02:05:31.755 INFO:tasks.ceph.mon.b:Stopped 2024-04-19T02:05:31.755 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-19T02:05:31.755 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T02:05:31.778 INFO:tasks.ceph.mon.c:Stopped 2024-04-19T02:05:31.778 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-19T02:05:31.778 DEBUG:teuthology.orchestra.run.smithi059:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | head -n 1 2024-04-19T02:05:31.797 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi059.front.sepia.ceph.com 2024-04-19T02:05:31.798 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-19T02:05:32.157 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi059.front.sepia.ceph.com 2024-04-19T02:05:32.158 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-19T02:05:32.354 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi059.front.sepia.ceph.com 2024-04-19T02:05:32.354 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-19T02:05:32.534 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi059.front.sepia.ceph.com 2024-04-19T02:05:32.534 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-19T02:05:32.711 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi081.front.sepia.ceph.com 2024-04-19T02:05:32.711 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-19T02:05:32.776 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.11 has been restored 2024-04-19T02:05:32.910 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi081.front.sepia.ceph.com 2024-04-19T02:05:32.910 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-19T02:05:33.122 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi081.front.sepia.ceph.com 2024-04-19T02:05:33.122 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-19T02:05:33.294 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi081.front.sepia.ceph.com 2024-04-19T02:05:33.294 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-19T02:05:33.472 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-8 on ubuntu@smithi179.front.sepia.ceph.com 2024-04-19T02:05:33.472 DEBUG:teuthology.orchestra.run.smithi179:> sync && sudo umount -f /var/lib/ceph/osd/ceph-8 2024-04-19T02:05:33.635 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-9 on ubuntu@smithi179.front.sepia.ceph.com 2024-04-19T02:05:33.635 DEBUG:teuthology.orchestra.run.smithi179:> sync && sudo umount -f /var/lib/ceph/osd/ceph-9 2024-04-19T02:05:33.776 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-10 on ubuntu@smithi179.front.sepia.ceph.com 2024-04-19T02:05:33.776 DEBUG:teuthology.orchestra.run.smithi179:> sync && sudo umount -f /var/lib/ceph/osd/ceph-10 2024-04-19T02:05:33.907 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-11 on ubuntu@smithi179.front.sepia.ceph.com 2024-04-19T02:05:33.907 DEBUG:teuthology.orchestra.run.smithi179:> sync && sudo umount -f /var/lib/ceph/osd/ceph-11 2024-04-19T02:05:34.044 INFO:tasks.ceph:Archiving mon data... 2024-04-19T02:05:34.045 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662749/data/mon.a.tgz 2024-04-19T02:05:34.045 DEBUG:teuthology.orchestra.run.smithi059:> mktemp 2024-04-19T02:05:34.051 INFO:teuthology.orchestra.run.smithi059.stdout:/tmp/tmp.rmk9CGkroK 2024-04-19T02:05:34.053 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.rmk9CGkroK 2024-04-19T02:05:34.173 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0666 /tmp/tmp.rmk9CGkroK 2024-04-19T02:05:34.241 DEBUG:teuthology.orchestra.remote:smithi059:/tmp/tmp.rmk9CGkroK is 298KB 2024-04-19T02:05:34.306 DEBUG:teuthology.orchestra.run.smithi059:> rm -fr /tmp/tmp.rmk9CGkroK 2024-04-19T02:05:34.312 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662749/data/mon.b.tgz 2024-04-19T02:05:34.312 DEBUG:teuthology.orchestra.run.smithi081:> mktemp 2024-04-19T02:05:34.318 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.iaKrnrYl2Y 2024-04-19T02:05:34.319 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.iaKrnrYl2Y 2024-04-19T02:05:34.464 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.iaKrnrYl2Y 2024-04-19T02:05:34.527 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.iaKrnrYl2Y is 402KB 2024-04-19T02:05:34.549 DEBUG:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.iaKrnrYl2Y 2024-04-19T02:05:34.554 DEBUG:teuthology.misc:Transferring archived files from smithi179:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662749/data/mon.c.tgz 2024-04-19T02:05:34.554 DEBUG:teuthology.orchestra.run.smithi179:> mktemp 2024-04-19T02:05:34.560 INFO:teuthology.orchestra.run.smithi179.stdout:/tmp/tmp.mJUOVkp08O 2024-04-19T02:05:34.560 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.mJUOVkp08O 2024-04-19T02:05:34.704 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 0666 /tmp/tmp.mJUOVkp08O 2024-04-19T02:05:34.766 DEBUG:teuthology.orchestra.remote:smithi179:/tmp/tmp.mJUOVkp08O is 402KB 2024-04-19T02:05:34.829 DEBUG:teuthology.orchestra.run.smithi179:> rm -fr /tmp/tmp.mJUOVkp08O 2024-04-19T02:05:34.835 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-19T02:05:34.836 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-19T02:05:34.838 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-19T02:05:34.841 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-19T02:05:34.901 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-19T02:05:34.907 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-19T02:05:34.939 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-19T02:05:34.940 INFO:tasks.ceph:Archiving crash dumps... 2024-04-19T02:05:34.941 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662749/remote/smithi059/crash 2024-04-19T02:05:34.942 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-19T02:05:34.959 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662749/remote/smithi081/crash 2024-04-19T02:05:34.960 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-19T02:05:34.973 DEBUG:teuthology.misc:Transferring archived files from smithi179:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662749/remote/smithi179/crash 2024-04-19T02:05:34.974 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-19T02:05:34.993 INFO:tasks.ceph:Compressing logs... 2024-04-19T02:05:34.994 DEBUG:teuthology.orchestra.run.smithi059:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-19T02:05:35.001 DEBUG:teuthology.orchestra.run.smithi081:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-19T02:05:35.020 DEBUG:teuthology.orchestra.run.smithi179:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-19T02:05:35.021 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34878.log 2024-04-19T02:05:35.021 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31632.log 2024-04-19T02:05:35.022 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36334.log 2024-04-19T02:05:35.022 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34878.log.gz 2024-04-19T02:05:35.022 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33061.log 2024-04-19T02:05:35.022 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31632.log.gz 2024-04-19T02:05:35.023 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36334.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36334.log.gz 2024-04-19T02:05:35.023 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32812.log 2024-04-19T02:05:35.023 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33061.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33061.log.gz 2024-04-19T02:05:35.025 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31155.log 2024-04-19T02:05:35.025 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30143.log 2024-04-19T02:05:35.025 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32812.log: /var/log/ceph/ceph-client.admin.31155.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32812.log.gz 2024-04-19T02:05:35.025 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31155.log.gz 2024-04-19T02:05:35.025 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35314.log 2024-04-19T02:05:35.025 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34148.log 2024-04-19T02:05:35.025 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30143.log.gz 2024-04-19T02:05:35.026 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35314.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35314.log.gz -5 2024-04-19T02:05:35.026 INFO:teuthology.orchestra.run.smithi059.stderr: --verbose -- /var/log/ceph/ceph-client.admin.34332.log 2024-04-19T02:05:35.028 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32869.log 2024-04-19T02:05:35.029 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34148.log.gz 2024-04-19T02:05:35.029 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34332.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34332.log.gz 2024-04-19T02:05:35.029 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29836.log 2024-04-19T02:05:35.029 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32869.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32869.log.gz 2024-04-19T02:05:35.029 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35060.log 2024-04-19T02:05:35.029 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31754.log 2024-04-19T02:05:35.029 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29836.log.gz 2024-04-19T02:05:35.029 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35060.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35060.log.gz 2024-04-19T02:05:35.029 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32537.log 2024-04-19T02:05:35.029 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32513.log 2024-04-19T02:05:35.029 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31754.log.gz 2024-04-19T02:05:35.029 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32467.log 2024-04-19T02:05:35.029 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32537.log.gz 2024-04-19T02:05:35.029 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32513.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32513.log.gz 2024-04-19T02:05:35.029 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35731.log 2024-04-19T02:05:35.032 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32467.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32467.log.gz 2024-04-19T02:05:35.032 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33036.log 2024-04-19T02:05:35.032 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35749.log 2024-04-19T02:05:35.032 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35731.log.gz 2024-04-19T02:05:35.032 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36516.log 2024-04-19T02:05:35.032 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33036.log.gz 2024-04-19T02:05:35.032 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35749.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35749.log.gzgzip 2024-04-19T02:05:35.032 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.29607.log 2024-04-19T02:05:35.032 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29766.log 2024-04-19T02:05:35.032 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36516.log.gz 2024-04-19T02:05:35.032 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29819.log 2024-04-19T02:05:35.032 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29607.log.gz 2024-04-19T02:05:35.033 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29766.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29766.log.gz 2024-04-19T02:05:35.033 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33313.log 2024-04-19T02:05:35.035 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36568.log 2024-04-19T02:05:35.035 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29819.log.gz 2024-04-19T02:05:35.036 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33313.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-19T02:05:35.036 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33313.log.gz 2024-04-19T02:05:35.036 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36568.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36568.log.gz 2024-04-19T02:05:35.036 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34904.log 2024-04-19T02:05:35.036 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29834.log 2024-04-19T02:05:35.036 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30624.log 2024-04-19T02:05:35.036 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34904.log.gz 2024-04-19T02:05:35.036 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29834.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29834.log.gz 2024-04-19T02:05:35.036 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34249.log 2024-04-19T02:05:35.036 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34362.log 2024-04-19T02:05:35.036 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30624.log.gz 2024-04-19T02:05:35.036 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34249.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34249.log.gz 2024-04-19T02:05:35.036 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36360.log 2024-04-19T02:05:35.036 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30137.log 2024-04-19T02:05:35.041 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34362.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.34540.log -- replaced with /var/log/ceph/ceph-client.admin.34362.log.gz 2024-04-19T02:05:35.041 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-04-19T02:05:35.042 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36360.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36360.log.gz 2024-04-19T02:05:35.042 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30860.log 2024-04-19T02:05:35.042 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30137.log.gz 2024-04-19T02:05:35.042 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34540.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34540.log.gz 2024-04-19T02:05:35.042 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32737.log 2024-04-19T02:05:35.042 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33987.log 2024-04-19T02:05:35.042 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30860.log.gz 2024-04-19T02:05:35.042 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36100.log 2024-04-19T02:05:35.042 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32737.log.gz 2024-04-19T02:05:35.042 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36308.log 2024-04-19T02:05:35.042 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33987.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33987.log.gz 2024-04-19T02:05:35.042 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31294.log 2024-04-19T02:05:35.042 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36100.log.gz 2024-04-19T02:05:35.042 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31557.log 2024-04-19T02:05:35.042 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36308.log.gz 2024-04-19T02:05:35.043 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35366.log 2024-04-19T02:05:35.043 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31294.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31294.log.gz 2024-04-19T02:05:35.043 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33837.log 2024-04-19T02:05:35.043 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31557.log.gz 2024-04-19T02:05:35.043 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31000.log 2024-04-19T02:05:35.043 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35366.log.gz 2024-04-19T02:05:35.043 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32137.log 2024-04-19T02:05:35.043 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33837.log.gz 2024-04-19T02:05:35.043 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33287.log 2024-04-19T02:05:35.043 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31000.log.gz 2024-04-19T02:05:35.043 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33926.log 2024-04-19T02:05:35.043 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32137.log.gz 2024-04-19T02:05:35.043 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33111.log 2024-04-19T02:05:35.043 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33287.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33287.log.gz 2024-04-19T02:05:35.049 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36490.log 2024-04-19T02:05:35.049 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33926.log.gz 2024-04-19T02:05:35.049 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33111.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33111.log.gz 2024-04-19T02:05:35.049 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36256.log 2024-04-19T02:05:35.049 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30623.log 2024-04-19T02:05:35.049 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36490.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36490.log.gz 2024-04-19T02:05:35.049 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36412.log 2024-04-19T02:05:35.049 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36256.log.gz 2024-04-19T02:05:35.049 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30623.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-19T02:05:35.049 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30623.log.gz 2024-04-19T02:05:35.049 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5/var/log/ceph/ceph-client.admin.36412.log: --verbose -- /var/log/ceph/ceph-client.admin.33641.log 2024-04-19T02:05:35.049 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36412.log.gz 2024-04-19T02:05:35.049 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35354.log 2024-04-19T02:05:35.050 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32032.log 2024-04-19T02:05:35.050 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33641.log.gz 2024-04-19T02:05:35.050 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36698.log 2024-04-19T02:05:35.050 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35354.log.gz 2024-04-19T02:05:35.050 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32403.log 2024-04-19T02:05:35.050 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32032.log.gz 2024-04-19T02:05:35.050 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36698.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36698.log.gz 2024-04-19T02:05:35.050 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33612.log 2024-04-19T02:05:35.050 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35384.log 2024-04-19T02:05:35.050 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32403.log.gz 2024-04-19T02:05:35.050 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33612.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33612.log.gz 2024-04-19T02:05:35.050 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-19T02:05:35.050 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32902.log 2024-04-19T02:05:35.050 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35384.log.gz 2024-04-19T02:05:35.051 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32185.log 2024-04-19T02:05:35.051 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32902.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32902.log.gz 2024-04-19T02:05:35.051 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30095.log 2024-04-19T02:05:35.051 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35695.log 2024-04-19T02:05:35.058 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-19T02:05:35.058 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-19T02:05:35.058 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-19T02:05:35.058 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-19T02:05:35.058 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-19T02:05:35.058 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-19T02:05:35.058 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-19T02:05:35.058 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-19T02:05:35.058 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-mon.b.log: 86.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-19T02:05:35.058 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.4.log: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-19T02:05:35.085 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25062.log 2024-04-19T02:05:35.085 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25086.log 2024-04-19T02:05:35.086 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.25062.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25062.log.gz 2024-04-19T02:05:35.086 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2024-04-19T02:05:35.086 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.25086.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25086.log.gz 2024-04-19T02:05:35.086 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25206.log 2024-04-19T02:05:35.086 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.11.log 2024-04-19T02:05:35.086 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-osd.8.log: /var/log/ceph/ceph-client.admin.25206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25038.log 2024-04-19T02:05:35.086 INFO:teuthology.orchestra.run.smithi179.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25206.log.gz 2024-04-19T02:05:35.086 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25278.log 2024-04-19T02:05:35.086 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.25038.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-19T02:05:35.086 INFO:teuthology.orchestra.run.smithi179.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25038.log.gz 2024-04-19T02:05:35.086 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.25278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25230.log 2024-04-19T02:05:35.086 INFO:teuthology.orchestra.run.smithi179.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25278.log.gz 2024-04-19T02:05:35.086 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25110.log 2024-04-19T02:05:35.087 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-19T02:05:35.087 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.25230.log: /var/log/ceph/ceph-client.admin.25110.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25230.log.gz 2024-04-19T02:05:35.087 INFO:teuthology.orchestra.run.smithi179.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25110.log.gz 2024-04-19T02:05:35.087 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25182.log 2024-04-19T02:05:35.087 INFO:teuthology.orchestra.run.smithi179.stderr: 86.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-19T02:05:35.087 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25254.log 2024-04-19T02:05:35.087 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.25182.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25182.log.gz 2024-04-19T02:05:35.087 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-19T02:05:35.087 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2024-04-19T02:05:35.087 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.25254.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25254.log.gz 2024-04-19T02:05:35.087 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25014.log 2024-04-19T02:05:35.087 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.10.log 2024-04-19T02:05:35.087 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.25014.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25014.log.gz 2024-04-19T02:05:35.088 INFO:teuthology.orchestra.run.smithi179.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.25158.log 2024-04-19T02:05:35.088 INFO:teuthology.orchestra.run.smithi179.stderr: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-19T02:05:35.088 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25134.log 2024-04-19T02:05:35.088 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.25158.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25158.log.gz 2024-04-19T02:05:35.088 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.25134.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25134.log.gz 2024-04-19T02:05:35.120 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32185.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32185.log.gz 2024-04-19T02:05:35.120 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31958.log 2024-04-19T02:05:35.120 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30095.log.gz 2024-04-19T02:05:35.120 INFO:teuthology.orchestra.run.smithi059.stderr:gzip/var/log/ceph/ceph-client.admin.35695.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30621.log 2024-04-19T02:05:35.120 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35695.log.gz 2024-04-19T02:05:35.120 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34670.log 2024-04-19T02:05:35.120 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31958.log.gz 2024-04-19T02:05:35.120 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30621.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.30621.log.gz --verbose 2024-04-19T02:05:35.120 INFO:teuthology.orchestra.run.smithi059.stderr: -- /var/log/ceph/ceph-client.admin.31902.log 2024-04-19T02:05:35.120 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36074.log 2024-04-19T02:05:35.120 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34670.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34670.log.gz 2024-04-19T02:05:35.121 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30117.log 2024-04-19T02:05:35.121 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31902.log.gz 2024-04-19T02:05:35.121 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33695.log 2024-04-19T02:05:35.121 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36074.log.gz 2024-04-19T02:05:35.121 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31180.log 2024-04-19T02:05:35.121 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30117.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30117.log.gz 2024-04-19T02:05:35.121 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33562.log 2024-04-19T02:05:35.121 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33695.log.gz 2024-04-19T02:05:35.121 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34047.log 2024-04-19T02:05:35.121 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31180.log.gz 2024-04-19T02:05:35.121 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31366.log 2024-04-19T02:05:35.121 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33562.log.gz 2024-04-19T02:05:35.121 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35062.log 2024-04-19T02:05:35.122 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34047.log.gz 2024-04-19T02:05:35.122 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36438.log 2024-04-19T02:05:35.122 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31366.log.gz 2024-04-19T02:05:35.122 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35062.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35062.log.gz 2024-04-19T02:05:35.122 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36464.log 2024-04-19T02:05:35.122 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36438.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36438.log.gz 2024-04-19T02:05:35.122 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35698.log 2024-04-19T02:05:35.122 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36464.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36464.log.gz -5 2024-04-19T02:05:35.122 INFO:teuthology.orchestra.run.smithi059.stderr: --verbose -- /var/log/ceph/ceph-client.admin.30126.log 2024-04-19T02:05:35.122 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34462.log 2024-04-19T02:05:35.122 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35698.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35698.log.gz 2024-04-19T02:05:35.122 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33362.log 2024-04-19T02:05:35.122 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30126.log.gz 2024-04-19T02:05:35.123 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34982.log 2024-04-19T02:05:35.123 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34462.log.gz 2024-04-19T02:05:35.123 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34414.log 2024-04-19T02:05:35.123 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33362.log.gz 2024-04-19T02:05:35.123 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30153.log 2024-04-19T02:05:35.123 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34982.log.gz 2024-04-19T02:05:35.123 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34275.log 2024-04-19T02:05:35.123 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34414.log.gz 2024-04-19T02:05:35.123 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31454.log 2024-04-19T02:05:35.123 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30153.log.gz 2024-04-19T02:05:35.123 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35078.log 2024-04-19T02:05:35.123 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34275.log.gz 2024-04-19T02:05:35.123 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.30959.log 2024-04-19T02:05:35.123 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31454.log.gz 2024-04-19T02:05:35.123 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31004.log 2024-04-19T02:05:35.124 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35078.log.gz 2024-04-19T02:05:35.124 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.30959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36724.log 2024-04-19T02:05:35.124 INFO:teuthology.orchestra.run.smithi059.stderr: 53.2% -- replaced with /var/log/ceph/ceph-client.0.30959.log.gz 2024-04-19T02:05:35.124 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35996.log 2024-04-19T02:05:35.124 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31004.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31004.log.gz 2024-04-19T02:05:35.124 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33776.log 2024-04-19T02:05:35.124 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36724.log.gz 2024-04-19T02:05:35.124 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24873.log 2024-04-19T02:05:35.124 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35996.log.gz 2024-04-19T02:05:35.124 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32210.log 2024-04-19T02:05:35.124 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33776.log.gz 2024-04-19T02:05:35.124 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.24873.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24873.log.gz 2024-04-19T02:05:35.124 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35666.log 2024-04-19T02:05:35.124 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35076.log 2024-04-19T02:05:35.125 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32210.log.gz 2024-04-19T02:05:35.125 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35666.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35666.log.gz 2024-04-19T02:05:35.125 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35365.log 2024-04-19T02:05:35.125 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36750.log 2024-04-19T02:05:35.125 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35076.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35076.log.gz 2024-04-19T02:05:35.125 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35365.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-19T02:05:35.125 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35365.log.gz 2024-04-19T02:05:35.125 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31865.log 2024-04-19T02:05:35.125 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36750.log.gz 2024-04-19T02:05:35.125 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34566.log 2024-04-19T02:05:35.125 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.31865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32058.log 2024-04-19T02:05:35.125 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31865.log.gz 2024-04-19T02:05:35.125 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34306.log 2024-04-19T02:05:35.126 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34566.log.gz 2024-04-19T02:05:35.126 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32058.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.32058.log.gz /var/log/ceph/ceph-client.admin.31254.log 2024-04-19T02:05:35.126 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-04-19T02:05:35.126 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29829.log 2024-04-19T02:05:35.126 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34306.log.gz 2024-04-19T02:05:35.126 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.31254.log: -- /var/log/ceph/ceph-client.admin.35077.log 2024-04-19T02:05:35.126 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31254.log.gz 2024-04-19T02:05:35.126 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34696.log 2024-04-19T02:05:35.126 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29829.log.gz 2024-04-19T02:05:35.126 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30494.log 2024-04-19T02:05:35.126 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35077.log.gz 2024-04-19T02:05:35.127 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34096.log 2024-04-19T02:05:35.127 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34696.log.gz 2024-04-19T02:05:35.127 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36230.log 2024-04-19T02:05:35.127 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30494.log.gz 2024-04-19T02:05:35.127 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35073.log 2024-04-19T02:05:35.127 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34096.log.gz 2024-04-19T02:05:35.127 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35662.log 2024-04-19T02:05:35.127 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36230.log.gz 2024-04-19T02:05:35.127 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36022.log 2024-04-19T02:05:35.127 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35073.log.gz 2024-04-19T02:05:35.127 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36152.log 2024-04-19T02:05:35.127 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35662.log.gz 2024-04-19T02:05:35.128 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36672.log 2024-04-19T02:05:35.128 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36022.log.gz 2024-04-19T02:05:35.128 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30416.log 2024-04-19T02:05:35.128 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36152.log.gz 2024-04-19T02:05:35.128 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36620.log 2024-04-19T02:05:35.128 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36672.log.gz 2024-04-19T02:05:35.128 INFO:teuthology.orchestra.run.smithi059.stderr:gzip/var/log/ceph/ceph-client.admin.30416.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.36594.log 2024-04-19T02:05:35.128 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30416.log.gz 2024-04-19T02:05:35.128 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32936.log 2024-04-19T02:05:35.128 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36620.log.gz 2024-04-19T02:05:35.128 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35391.log 2024-04-19T02:05:35.128 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36594.log.gz 2024-04-19T02:05:35.129 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30617.log 2024-04-19T02:05:35.129 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32936.log.gz 2024-04-19T02:05:35.129 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35391.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-19T02:05:35.129 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35391.log.gz 2024-04-19T02:05:35.129 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30771.log 2024-04-19T02:05:35.129 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30617.log.gz 2024-04-19T02:05:35.129 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29714.log 2024-04-19T02:05:35.129 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31963.log 2024-04-19T02:05:35.129 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30771.log.gz 2024-04-19T02:05:35.129 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32965.log 2024-04-19T02:05:35.129 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29714.log.gz 2024-04-19T02:05:35.129 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31963.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31963.log.gz 2024-04-19T02:05:35.129 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30546.log 2024-04-19T02:05:35.129 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35008.log 2024-04-19T02:05:35.129 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32965.log.gz 2024-04-19T02:05:35.130 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34930.log 2024-04-19T02:05:35.130 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30546.log.gz 2024-04-19T02:05:35.130 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33142.log 2024-04-19T02:05:35.130 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35008.log: /var/log/ceph/ceph-client.admin.34930.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35008.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31729.log 2024-04-19T02:05:35.130 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% 2024-04-19T02:05:35.130 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34930.log.gz 2024-04-19T02:05:35.130 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33142.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33142.log.gz 2024-04-19T02:05:35.130 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34618.log 2024-04-19T02:05:35.130 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35970.log 2024-04-19T02:05:35.130 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31729.log.gz 2024-04-19T02:05:35.130 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34618.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.34618.log.gz 2024-04-19T02:05:35.130 INFO:teuthology.orchestra.run.smithi059.stderr: /var/log/ceph/ceph-client.admin.31405.log 2024-04-19T02:05:35.130 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30810.log 2024-04-19T02:05:35.130 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35970.log.gz 2024-04-19T02:05:35.130 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30468.log 2024-04-19T02:05:35.131 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31405.log.gz 2024-04-19T02:05:35.131 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35371.log 2024-04-19T02:05:35.131 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30810.log.gz 2024-04-19T02:05:35.131 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34644.log 2024-04-19T02:05:35.131 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30468.log.gz 2024-04-19T02:05:35.131 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35371.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35371.log.gz 2024-04-19T02:05:35.131 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25067.log 2024-04-19T02:05:35.131 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34826.log 2024-04-19T02:05:35.132 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34644.log.gz 2024-04-19T02:05:35.132 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.25067.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34852.log 2024-04-19T02:05:35.132 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.25067.log.gz 2024-04-19T02:05:35.132 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33957.log 2024-04-19T02:05:35.132 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34826.log.gz 2024-04-19T02:05:35.132 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34358.log 2024-04-19T02:05:35.132 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34852.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34852.log.gz 2024-04-19T02:05:35.132 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29740.log 2024-04-19T02:05:35.132 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33957.log.gz 2024-04-19T02:05:35.132 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33867.log 2024-04-19T02:05:35.132 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34358.log: /var/log/ceph/ceph-client.admin.29740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29818.log 2024-04-19T02:05:35.133 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29740.log.gz 2024-04-19T02:05:35.133 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34358.log.gz 2024-04-19T02:05:35.133 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33867.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33867.log.gz 2024-04-19T02:05:35.133 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20129.log 2024-04-19T02:05:35.133 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29818.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-19T02:05:35.133 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29818.log.gz 2024-04-19T02:05:35.133 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph.tmp-client.admin.20129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29636.log 2024-04-19T02:05:35.133 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20129.log.gz 2024-04-19T02:05:35.133 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph.log: 86.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-19T02:05:35.133 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33806.log 2024-04-19T02:05:35.133 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35049.log 2024-04-19T02:05:35.133 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29636.log.gz 2024-04-19T02:05:35.133 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36126.log 2024-04-19T02:05:35.133 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33806.log.gz 2024-04-19T02:05:35.133 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35394.log 2024-04-19T02:05:35.134 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35049.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35049.log.gz 2024-04-19T02:05:35.134 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36126.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36126.log.gz 2024-04-19T02:05:35.134 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36204.log 2024-04-19T02:05:35.134 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31561.log 2024-04-19T02:05:35.134 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35394.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35394.log.gz 2024-04-19T02:05:35.134 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35727.log 2024-04-19T02:05:35.134 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36204.log.gz 2024-04-19T02:05:35.134 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32431.log 2024-04-19T02:05:35.134 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31561.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31561.log.gz 2024-04-19T02:05:35.134 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35059.log 2024-04-19T02:05:35.135 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35727.log.gz 2024-04-19T02:05:35.135 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34198.log 2024-04-19T02:05:35.135 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32431.log.gz 2024-04-19T02:05:35.135 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32132.log 2024-04-19T02:05:35.135 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35059.log.gz 2024-04-19T02:05:35.135 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35057.log 2024-04-19T02:05:35.135 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34198.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34198.log.gz 2024-04-19T02:05:35.135 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32599.log 2024-04-19T02:05:35.135 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32132.log.gz 2024-04-19T02:05:35.135 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35057.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35057.log.gz 2024-04-19T02:05:35.135 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35072.log 2024-04-19T02:05:35.135 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36048.log 2024-04-19T02:05:35.135 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32599.log.gz 2024-04-19T02:05:35.136 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35072.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33643.log 2024-04-19T02:05:35.136 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35072.log.gz 2024-04-19T02:05:35.136 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34722.log 2024-04-19T02:05:35.136 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36048.log.gz 2024-04-19T02:05:35.136 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35722.log 2024-04-19T02:05:35.136 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33643.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33643.log.gz 2024-04-19T02:05:35.136 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30098.log 2024-04-19T02:05:35.136 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34722.log.gz 2024-04-19T02:05:35.136 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29682.log 2024-04-19T02:05:35.136 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35722.log: /var/log/ceph/ceph-client.admin.30098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31633.log 2024-04-19T02:05:35.136 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35722.log.gz 2024-04-19T02:05:35.136 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30098.log.gz 2024-04-19T02:05:35.136 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29682.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29682.log.gz 2024-04-19T02:05:35.137 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33366.log 2024-04-19T02:05:35.137 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30865.log 2024-04-19T02:05:35.137 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31633.log.gz 2024-04-19T02:05:35.137 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32062.log 2024-04-19T02:05:35.137 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33366.log.gz 2024-04-19T02:05:35.137 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35748.log 2024-04-19T02:05:35.137 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30865.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30865.log.gz 2024-04-19T02:05:35.137 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30576.log 2024-04-19T02:05:35.137 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32062.log.gz 2024-04-19T02:05:35.137 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35748.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35748.log.gz 2024-04-19T02:05:35.137 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32319.log 2024-04-19T02:05:35.137 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35075.log 2024-04-19T02:05:35.137 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30576.log.gz 2024-04-19T02:05:35.138 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31059.log 2024-04-19T02:05:35.138 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32319.log.gz 2024-04-19T02:05:35.138 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32703.log 2024-04-19T02:05:35.138 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35075.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35075.log.gz 2024-04-19T02:05:35.138 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34174.log 2024-04-19T02:05:35.138 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31059.log.gz 2024-04-19T02:05:35.138 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36386.log 2024-04-19T02:05:35.138 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32703.log: /var/log/ceph/ceph-client.admin.34174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29821.log 2024-04-19T02:05:35.138 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32703.log.gz 2024-04-19T02:05:35.138 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34174.log.gz/var/log/ceph/ceph-client.admin.36386.log: 2024-04-19T02:05:35.138 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31704.log 2024-04-19T02:05:35.138 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36386.log.gz 2024-04-19T02:05:35.139 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29821.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29821.log.gz 2024-04-19T02:05:35.139 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36779.log 2024-04-19T02:05:35.139 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33742.log 2024-04-19T02:05:35.139 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31704.log.gz 2024-04-19T02:05:35.139 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32260.log 2024-04-19T02:05:35.139 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36779.log.gz 2024-04-19T02:05:35.139 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33742.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33742.log.gz 2024-04-19T02:05:35.139 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29823.log 2024-04-19T02:05:35.139 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24959.log 2024-04-19T02:05:35.139 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32260.log.gz 2024-04-19T02:05:35.139 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29823.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29823.log.gz 2024-04-19T02:05:35.139 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-19T02:05:35.139 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31761.log 2024-04-19T02:05:35.139 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.24959.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24959.log.gz/var/log/ceph/ceph.audit.log: 2024-04-19T02:05:35.139 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31480.log 2024-04-19T02:05:35.140 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32643.log 2024-04-19T02:05:35.140 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31761.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31761.log.gz 2024-04-19T02:05:35.140 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31480.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30173.log 2024-04-19T02:05:35.140 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31480.log.gz 2024-04-19T02:05:35.140 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32643.log: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-19T02:05:35.140 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32643.log.gz 2024-04-19T02:05:35.140 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30114.log 2024-04-19T02:05:35.140 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34413.log 2024-04-19T02:05:35.140 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30173.log.gz 2024-04-19T02:05:35.140 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31297.log 2024-04-19T02:05:35.140 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30114.log.gz 2024-04-19T02:05:35.140 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34413.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34413.log.gz 2024-04-19T02:05:35.140 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35353.log 2024-04-19T02:05:35.141 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36282.log 2024-04-19T02:05:35.141 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31297.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31297.log.gz 2024-04-19T02:05:35.141 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30141.log 2024-04-19T02:05:35.141 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35353.log.gz 2024-04-19T02:05:35.141 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29814.log 2024-04-19T02:05:35.141 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36282.log: /var/log/ceph/ceph-client.admin.30141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30121.log 2024-04-19T02:05:35.141 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36282.log.gz 2024-04-19T02:05:35.141 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30141.log.gz 2024-04-19T02:05:35.141 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35345.log 2024-04-19T02:05:35.141 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29814.log.gz 2024-04-19T02:05:35.141 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34017.log 2024-04-19T02:05:35.141 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30121.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30121.log.gz 2024-04-19T02:05:35.141 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33421.log 2024-04-19T02:05:35.142 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35345.log.gz 2024-04-19T02:05:35.142 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29813.log 2024-04-19T02:05:35.142 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34017.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34017.log.gz/var/log/ceph/ceph-client.admin.33421.log: 2024-04-19T02:05:35.142 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34514.log 2024-04-19T02:05:35.142 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33421.log.gz 2024-04-19T02:05:35.142 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34592.log 2024-04-19T02:05:35.142 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29813.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29813.log.gz 2024-04-19T02:05:35.142 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34748.log 2024-04-19T02:05:35.142 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34514.log.gz 2024-04-19T02:05:35.142 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33896.log 2024-04-19T02:05:35.142 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34592.log: 0.0%/var/log/ceph/ceph-client.admin.34748.log: -- replaced with /var/log/ceph/ceph-client.admin.34592.log.gzgzip 2024-04-19T02:05:35.143 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.34488.log 2024-04-19T02:05:35.143 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34748.log.gz 2024-04-19T02:05:35.143 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33896.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33896.log.gz 2024-04-19T02:05:35.143 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35754.log 2024-04-19T02:05:35.143 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35055.log 2024-04-19T02:05:35.143 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34488.log.gz 2024-04-19T02:05:35.143 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32310.log 2024-04-19T02:05:35.143 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35754.log.gz 2024-04-19T02:05:35.143 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35055.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35055.log.gz 2024-04-19T02:05:35.143 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29832.log 2024-04-19T02:05:35.143 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32368.log 2024-04-19T02:05:35.143 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32310.log.gz 2024-04-19T02:05:35.144 INFO:teuthology.orchestra.run.smithi081.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-19T02:05:35.153 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31831.log 2024-04-19T02:05:35.153 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29832.log.gz 2024-04-19T02:05:35.154 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.32368.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32368.log.gz 2024-04-19T02:05:35.154 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29824.log 2024-04-19T02:05:35.154 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30520.log 2024-04-19T02:05:35.154 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31831.log.gz 2024-04-19T02:05:35.154 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34774.log 2024-04-19T02:05:35.154 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29824.log.gz 2024-04-19T02:05:35.154 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34800.log 2024-04-19T02:05:35.154 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30520.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30520.log.gz 2024-04-19T02:05:35.154 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30913.log 2024-04-19T02:05:35.154 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34774.log.gz 2024-04-19T02:05:35.154 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34800.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34800.log.gz 2024-04-19T02:05:35.154 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36178.log 2024-04-19T02:05:35.154 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30913.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30913.log.gz 2024-04-19T02:05:35.154 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35658.log 2024-04-19T02:05:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35387.log 2024-04-19T02:05:35.155 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36178.log.gz 2024-04-19T02:05:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34122.log 2024-04-19T02:05:35.155 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35658.log.gz 2024-04-19T02:05:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36542.log 2024-04-19T02:05:35.155 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35387.log.gz 2024-04-19T02:05:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33537.log 2024-04-19T02:05:35.155 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34122.log.gz 2024-04-19T02:05:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35696.log 2024-04-19T02:05:35.155 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36542.log.gz 2024-04-19T02:05:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35379.log 2024-04-19T02:05:35.155 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33537.log.gz 2024-04-19T02:05:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33238.log 2024-04-19T02:05:35.155 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35696.log.gz 2024-04-19T02:05:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.35379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34956.log 2024-04-19T02:05:35.156 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35379.log.gz 2024-04-19T02:05:35.156 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30442.log 2024-04-19T02:05:35.156 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33238.log.gz 2024-04-19T02:05:35.156 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.34956.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34956.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33464.log 2024-04-19T02:05:35.156 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-04-19T02:05:35.156 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29822.log 2024-04-19T02:05:35.156 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30442.log.gz 2024-04-19T02:05:35.156 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36803.log 2024-04-19T02:05:35.156 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.33464.log: /var/log/ceph/ceph-client.admin.29822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31102.log 2024-04-19T02:05:35.156 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33464.log.gz 2024-04-19T02:05:35.156 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29822.log.gz 2024-04-19T02:05:35.156 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36803.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30129.log 2024-04-19T02:05:35.156 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36803.log.gz 2024-04-19T02:05:35.156 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.31102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32786.log 2024-04-19T02:05:35.157 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31102.log.gz 2024-04-19T02:05:35.157 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36646.log 2024-04-19T02:05:35.157 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.30129.log: /var/log/ceph/ceph-client.admin.32786.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30129.log.gz 2024-04-19T02:05:35.157 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32786.log.gz 2024-04-19T02:05:35.157 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.36646.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36646.log.gz 2024-04-19T02:05:35.182 INFO:teuthology.orchestra.run.smithi081.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-19T02:05:35.188 INFO:teuthology.orchestra.run.smithi179.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-19T02:05:35.267 INFO:teuthology.orchestra.run.smithi059.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-19T02:05:35.339 INFO:teuthology.orchestra.run.smithi081.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-19T02:05:35.533 INFO:teuthology.orchestra.run.smithi179.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.10.log.gz 2024-04-19T02:05:35.539 INFO:teuthology.orchestra.run.smithi081.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-19T02:05:35.618 INFO:teuthology.orchestra.run.smithi081.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-19T02:05:35.623 INFO:teuthology.orchestra.run.smithi179.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2024-04-19T02:05:35.814 INFO:teuthology.orchestra.run.smithi059.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-19T02:05:35.836 INFO:teuthology.orchestra.run.smithi059.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-19T02:05:35.962 INFO:teuthology.orchestra.run.smithi081.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-19T02:05:35.964 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-04-19T02:05:35.964 INFO:teuthology.orchestra.run.smithi081.stderr:real 0m0.942s 2024-04-19T02:05:35.964 INFO:teuthology.orchestra.run.smithi081.stderr:user 0m2.399s 2024-04-19T02:05:35.964 INFO:teuthology.orchestra.run.smithi081.stderr:sys 0m0.100s 2024-04-19T02:05:36.027 INFO:teuthology.orchestra.run.smithi059.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-19T02:05:36.029 INFO:teuthology.orchestra.run.smithi179.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2024-04-19T02:05:36.059 INFO:teuthology.orchestra.run.smithi179.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.11.log.gz 2024-04-19T02:05:36.060 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-04-19T02:05:36.060 INFO:teuthology.orchestra.run.smithi179.stderr:real 0m1.013s 2024-04-19T02:05:36.060 INFO:teuthology.orchestra.run.smithi179.stderr:user 0m3.010s 2024-04-19T02:05:36.060 INFO:teuthology.orchestra.run.smithi179.stderr:sys 0m0.100s 2024-04-19T02:05:36.064 INFO:teuthology.orchestra.run.smithi059.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-19T02:05:36.066 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-04-19T02:05:36.066 INFO:teuthology.orchestra.run.smithi059.stderr:real 0m1.062s 2024-04-19T02:05:36.066 INFO:teuthology.orchestra.run.smithi059.stderr:user 0m3.784s 2024-04-19T02:05:36.066 INFO:teuthology.orchestra.run.smithi059.stderr:sys 0m0.224s 2024-04-19T02:05:36.066 INFO:tasks.ceph:Archiving logs... 2024-04-19T02:05:36.066 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662749/remote/smithi059/log 2024-04-19T02:05:36.067 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-19T02:05:36.779 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662749/remote/smithi081/log 2024-04-19T02:05:36.780 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-19T02:05:37.126 DEBUG:teuthology.misc:Transferring archived files from smithi179:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662749/remote/smithi179/log 2024-04-19T02:05:37.127 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-19T02:05:37.566 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-19T02:05:37.577 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-19T02:05:37.577 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-19T02:05:37.580 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-19T02:05:37.582 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-19T02:05:37.682 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-19T02:05:37.692 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-19T02:05:37.692 DEBUG:teuthology.orchestra.run.smithi059:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-19T02:05:37.695 DEBUG:teuthology.orchestra.run.smithi081:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-19T02:05:37.698 DEBUG:teuthology.orchestra.run.smithi179:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-19T02:05:37.707 INFO:teuthology.orchestra.run.smithi059.stdout: remote refid st t when poll reach delay offset jitter 2024-04-19T02:05:37.707 INFO:teuthology.orchestra.run.smithi059.stdout:============================================================================== 2024-04-19T02:05:37.707 INFO:teuthology.orchestra.run.smithi059.stdout:+hv01.front.sepi 67.205.162.81 3 u 10 64 77 0.053 -2.226 0.688 2024-04-19T02:05:37.707 INFO:teuthology.orchestra.run.smithi059.stdout:*hv02.front.sepi 63.231.80.2 3 u 12 64 77 0.136 +0.966 1.251 2024-04-19T02:05:37.708 INFO:teuthology.orchestra.run.smithi059.stdout:+hv03.front.sepi 74.6.168.72 3 u 10 64 77 0.106 -2.280 1.101 2024-04-19T02:05:37.708 INFO:teuthology.orchestra.run.smithi059.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T02:05:37.709 INFO:teuthology.orchestra.run.smithi081.stdout: remote refid st t when poll reach delay offset jitter 2024-04-19T02:05:37.709 INFO:teuthology.orchestra.run.smithi081.stdout:============================================================================== 2024-04-19T02:05:37.709 INFO:teuthology.orchestra.run.smithi081.stdout:+hv01.front.sepi 67.205.162.81 3 u 13 64 77 0.080 -1.288 0.673 2024-04-19T02:05:37.709 INFO:teuthology.orchestra.run.smithi081.stdout:+hv02.front.sepi 63.231.80.2 3 u 7 64 77 0.090 -0.183 1.021 2024-04-19T02:05:37.709 INFO:teuthology.orchestra.run.smithi081.stdout:*hv03.front.sepi 74.6.168.72 3 u 10 64 77 0.089 -1.941 1.088 2024-04-19T02:05:37.709 INFO:teuthology.orchestra.run.smithi081.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T02:05:37.710 INFO:teuthology.orchestra.run.smithi179.stdout: remote refid st t when poll reach delay offset jitter 2024-04-19T02:05:37.711 INFO:teuthology.orchestra.run.smithi179.stdout:============================================================================== 2024-04-19T02:05:37.711 INFO:teuthology.orchestra.run.smithi179.stdout:+hv01.front.sepi 67.205.162.81 3 u 7 64 77 0.115 -1.781 0.894 2024-04-19T02:05:37.711 INFO:teuthology.orchestra.run.smithi179.stdout:+hv02.front.sepi 63.231.80.2 3 u 10 64 77 0.068 +0.372 0.894 2024-04-19T02:05:37.711 INFO:teuthology.orchestra.run.smithi179.stdout:*hv03.front.sepi 74.6.168.72 3 u 7 64 77 0.083 -1.118 0.812 2024-04-19T02:05:37.711 INFO:teuthology.orchestra.run.smithi179.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T02:05:37.712 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-19T02:05:37.721 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-19T02:05:37.721 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-19T02:05:37.882 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-19T02:05:37.892 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-19T02:05:37.900 INFO:teuthology.task.internal:Duration was 773.836828 seconds 2024-04-19T02:05:37.901 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-19T02:05:37.910 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-19T02:05:37.910 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-19T02:05:37.913 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-19T02:05:37.915 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-19T02:05:37.953 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-19T02:05:37.953 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi059.front.sepia.ceph.com 2024-04-19T02:05:37.954 DEBUG:teuthology.orchestra.run.smithi059:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-19T02:05:38.001 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi081.front.sepia.ceph.com 2024-04-19T02:05:38.001 DEBUG:teuthology.orchestra.run.smithi081:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-19T02:05:38.011 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi179.front.sepia.ceph.com 2024-04-19T02:05:38.011 DEBUG:teuthology.orchestra.run.smithi179:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-19T02:05:38.021 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-19T02:05:38.022 DEBUG:teuthology.orchestra.run.smithi059:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-19T02:05:38.045 DEBUG:teuthology.orchestra.run.smithi081:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-19T02:05:38.056 DEBUG:teuthology.orchestra.run.smithi179:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-19T02:05:38.086 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-19T02:05:38.087 DEBUG:teuthology.orchestra.run.smithi059:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-19T02:05:38.117 DEBUG:teuthology.orchestra.run.smithi081:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-19T02:05:38.124 DEBUG:teuthology.orchestra.run.smithi179:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-19T02:05:38.183 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-19T02:05:38.195 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-19T02:05:38.196 DEBUG:teuthology.orchestra.run.smithi059:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-19T02:05:38.209 DEBUG:teuthology.orchestra.run.smithi081:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-19T02:05:38.220 DEBUG:teuthology.orchestra.run.smithi179:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-19T02:05:38.231 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-19T02:05:38.241 DEBUG:teuthology.orchestra.run.smithi059:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-19T02:05:38.253 DEBUG:teuthology.orchestra.run.smithi081:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-19T02:05:38.262 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = core 2024-04-19T02:05:38.264 DEBUG:teuthology.orchestra.run.smithi179:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-19T02:05:38.273 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = core 2024-04-19T02:05:38.284 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern = core 2024-04-19T02:05:38.306 DEBUG:teuthology.orchestra.run.smithi059:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-19T02:05:38.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T02:05:38.336 DEBUG:teuthology.orchestra.run.smithi081:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-19T02:05:38.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T02:05:38.346 DEBUG:teuthology.orchestra.run.smithi179:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-19T02:05:38.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T02:05:38.355 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-19T02:05:38.365 INFO:teuthology.task.internal:Transferring archived files... 2024-04-19T02:05:38.366 DEBUG:teuthology.misc:Transferring archived files from smithi059:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662749/remote/smithi059 2024-04-19T02:05:38.366 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-19T02:05:38.400 DEBUG:teuthology.misc:Transferring archived files from smithi081:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662749/remote/smithi081 2024-04-19T02:05:38.400 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-19T02:05:38.421 DEBUG:teuthology.misc:Transferring archived files from smithi179:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-18_21:00:02-rados-squid-distro-default-smithi/7662749/remote/smithi179 2024-04-19T02:05:38.422 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-19T02:05:38.444 INFO:teuthology.task.internal:Removing archive directory... 2024-04-19T02:05:38.444 DEBUG:teuthology.orchestra.run.smithi059:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-19T02:05:38.446 DEBUG:teuthology.orchestra.run.smithi081:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-19T02:05:38.460 DEBUG:teuthology.orchestra.run.smithi179:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-19T02:05:38.487 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-19T02:05:38.499 INFO:teuthology.task.internal:Not uploading archives. 2024-04-19T02:05:38.499 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-19T02:05:38.508 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-19T02:05:38.508 DEBUG:teuthology.orchestra.run.smithi059:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-19T02:05:38.511 DEBUG:teuthology.orchestra.run.smithi081:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-19T02:05:38.513 DEBUG:teuthology.orchestra.run.smithi179:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-19T02:05:38.514 INFO:teuthology.orchestra.run.smithi059.stdout: 658639 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 19 02:05 /home/ubuntu/cephtest 2024-04-19T02:05:38.516 INFO:teuthology.orchestra.run.smithi081.stdout: 655429 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 19 02:05 /home/ubuntu/cephtest 2024-04-19T02:05:38.535 INFO:teuthology.orchestra.run.smithi179.stdout: 658644 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 19 02:05 /home/ubuntu/cephtest 2024-04-19T02:05:38.537 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-19T02:05:38.548 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-19T02:05:38.575 INFO:teuthology.run:Summary data: description: rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} duration: 773.8368284702301 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-04-19T02:05:38.575 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-19T02:05:38.658 INFO:teuthology.run:pass